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5th  Annual  CMMI  Technology  Conference  and  User  Group 
Denver,  CO 
14-17  November  2005 


Agenda 


Monday.  14  November  2005 


Tutorial  Tracks 
Track  1 : 

•  Calculating  CMMI-Based  Return  on  Investment  (ROI):  Why,  When,  What,  How?,  Mr.  Rolf  W.  Reitzig,  Cognece,  Inc. 

•  A  Practical  Guide  to  Implementing  Levels  4  and  5,  Mr.  Rick  Hefner,  Northrop  Grumman  Corporation 

Track  2: 

•  Agile/Lean  Workshop,  Mr.  Jeffrey  Dutton,  Jacobs  Sverdrup 

•  Mr.  Tim  Kasse,  Kasse  Initiatives,  LLC: 

1 .  Leveraging  ITIL  Services  (Suppport  and  Delivery)  Capability  and  Maturity  with  the  CMMI 

2.  Service  Management  “A  Process  Led  Approach” 

3.  ITIL  IT  Infrastructure  Library  Overview 

4.  Overview  of  Service  Support  &  Service  Delivery  Functions 

5.  Configuration  Management  &  Change  Management 

6.  “Service  Support”  Change  Management 

7.  “Service  Support”  Configuration  Management 

8.  Configuration  Management  &  Change  Management  ITIL  -  CMMI 

9.  ITIL  Process  Maturity  Self-Assessment  &  Action  Plan 

Track  3: 

•  The  Look  and  Feel  of  a  Successful  CMMI  Implementaiton,  Mr.  Tim  Kasse,  Kasse  Initiatives,  LLC 

•  How  to  Define  CMMI  Based  Process  That  are  Short  and  Usable  and  Using  a  Process  Measurement  Framework  to  Successfully  Achieve  Measurable  Results, 
Mr.  Timothy  G.  Olson,  Quality  Improvement  Consultatnts,  Inc. 

Track  4: 

•  Using  Simulation  to  Support  Better  Management  Decisions,  Dr.  David  M.  Raffo,  Portland  State  University 

•  Institutionalizing  Resource  Planning  and  Management  Part  I  and  Part  II,  Mr.  Donald  A.  Borcherding,  NexSummit,  LLC 


Track  5: 

•  The  CMMI  V1.2  ...  A  Tutorial,  Mr.  David  M.  Phillips,  SEI 


Track  6: 

•  Integrated  Porject  Management  (IPM)  -  The  CMMI  and  Collaborative  Product  Develop  and  Requirement  Engineering:  A  Practicial  Approach  to  Modeling 
and  Managing  Requirements,  Mr.  William  J.  Deibler,  II  Software  Systems  Quality  Consulting  -  SSQC 

Tuesday.  15  November  2005 

General  Sessions 

LTG  Joseph  Yakovac,  USA,  Miliatry  Deputy  Office  of  the  Secretary  of  the  Army,  Acquisition,  Logistics  &  Technology 
Executive  Panel:  "How  Has  CMMI  Improved  Our  Program  &  Project  Performance  —  Or  Has  It?": 
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Mr.  Mark  Schaeffer,  Director,  Systems  Engineering,  OUSD(AT&L)  Defense  System  and  OSD  Sponsor,  CMMI 

•  Mr.  Dev  Banerjee,  Division  Director,  Systems  &  Flight  Engineering,  Boeing  Integrated  Defense  Systems 

•  Mr.  John  Evers,  Raytheon  Processes  Program  Manager,  Raytheon  Common  Engineering  Process  Program 

•  Brig  Gen  Gary  Salisbury,  USAF  (Ret),  Executive  Director  Business  Development,  Defense  Mission  Systems,  Northrop  Grumman  Mission  Systems 
Lunch:  CMMI  State  of  the  Model,  Mr.  Bob  Rassa,  Raytheon;  Mr.  Clyde  Chittister,  SEI 

Technical  Sessions 

Track  1 :  CMMI  Process  Improvement 

•  CMMI/ISO  -  "Can't  we  all  just  get  along?",  Mr.  Dale  R.  Spaulding,  The  Boeing  Company 

•  Real  World  Application  of  IEEE  Software  Engineering  Standards  to  CMM/CMMI  Software  Process  Improvement  Initiatives,  Ms.  Susan  K.  Land,  Norhtrop 
Gurmman  IT/TASC 

•  The  CMMI  Product  Suite  and  International  Standards  —  An  Update,  Mr.  David  H.  Kitson,  SEI 
Track  2:  Practical  Guidance 

•  Verification  in  CMMI  using  Peer  Reviews  -  Presentation  and  Paper,  Ms.  Jeanne  H.  Balsam,  Georgia  Tech  Research  Institute 

•  Process  QA  in  the  Information  Age:  Keep  it  Light!,  Hillel  Glazer,  Entinex,  Inc 

•  Defect  Datat  and  Configuation  Management,  Ms.  Julie  E.  Schmarje,  Raytheon  Company 

Track  3:  Appraisals 

•  Wasted  Days  and  Wasted  Nights  -  Leveraging  Your  Appraisal  Team  as  a  Resource,  Dr.  Timothy  J.  Davis,  Raytheon  Missile  Systems 

•  Building  a  Credible  SCAMPI  Appraisal  Representative  Sample,  Mr.  Robert  L.  Moore,  III.,  Business  Transformation  Institute,  Inc. 

•  Top  10  Signs  You're  Ready  (or  Not)  for  an  Appraisal,  Mr.  Gary  Natwick,  Harris  Corporation 

Track  4:  ROI  &  Benefits  of  CMMI 

•  Measuring  Performance:  Evidence  About  the  Results  of  CMMI,  Ms.  Diane  Gibson,  SEI 

•  Prioritizing  Process  Improvement  Strategies  in  CMMI  to  Optimize  Business  Objectives,  Dr.  Aldo  Dagnino,  ABB,  Inc.  US  Corporate  Research 

•  Implementing  a  Plan  for  Controlling  ROI  for  CMMI  Process  Improvement,  Mr.  J.  M.  Perry,  BAE  Systems 

•  Lessons  Learned  in  the  Engineering  of  Process  Performance  Models  on  the  Journey  to  Higher  Maturity  Levels,  Mr.  Dr.  Mary  Anne  Herndon,  SAIC 
Track  5:  Acquisition  /  High  Maturity 

•  Getting  Lost  on  the  Way  to  Level  5,  Ms.  Kathy  King,  The  Center  for  Systems  Management 

•  Understanding  Why?,  Mr.  David  N.  Card,  Q-Labs 

Track  6:  Transitioning  to  CMMI 

•  Migrating  Best  Practices  Within  an  Organization:  Experiences  in  Adapting  CMMI  Policies  and  Procedures  Used  in  One  Part  of  a  Business  to  Another,  Mr. 
Scott  Sherrill,  Georgia  Tech  Research  Institute 

•  An  Enterprise  Wide  CMMI  Implementation  at  Accenture,  Ms.  Sarah  S.  Bengzon,  Accenture 

•  Stakeholder  Identification  and  Involvement  in  the  CMMI,  Mr.  James  R.  Armstrong,  Systems  and  Software  Consortium 

•  Ensuring  the  Right  Process  is  Deployed  Right:  Synchronizing  Process  Checkpoints  with  Business  Rhythm,  Ms.  Joan  Weszka,  Lockheed  Martin  Corporation 
Track  7:  CMMI  for  Small  Projects  anhd  Organizations 

•  Making  PPQA  Work  on  Small  Projects  Presentation  and  Paper,  Ms.  Jean  M.  Swank,  Georgia  Tech  Research  Institute 

•  Does  Size  Matter  in  CMMI  Implementation  or  Was  Yoda  Wrong?,  Mr.  Paul  H.  Meyers,  SAIC 


Wednesday.  16  November  2005 


Technical  Sessions 


Track  1:  CMMI  Process  Improvement 

•  A  Change  Agent  in  a  Level  1  Organization;  How  to  Survive  in  a  Hostile  Environment,  Mr.  Andrew  Cordes,  ABB  -  United  States  Corporate  Research  Center 

•  "Sound  Systems  Engineering  Using  CMMI,  Mr.  Michael  T.  Kutch,  Jr.,  SPAWAR  -  Charleston 

•  Using  CMMI  to  "Dig  Out"  from  an  Ad  Hoc  Development?,  Mr.  Donald  A.  Borcherding,  NexSummit,  LLC 

•  Strategic  Planning:  Selling  a  CMMI-Based  Improvement  Effort  to  Senior  Management,  Dr.  Aldo  Dagnino,  ABB,  Inc.,  US  Corporate  Research 

•  Enterprise  Process  Intergration  within  the  Space  and  Airborne  Systems  Business  Area  of  Raytheon,  Mrs.  Deana  A.  Seigler,  Raytheon  Company 

•  Interpreting  the  CMMI:  It  Depends!,  Mr.  Rick  Hefner,  Northrop  Grumman  Corporation 

•  CMMI  as  Safeguard  Against  Software  Entropy:  A  Manager's  Perspective,  Dr.  Thomas  F.  Christian,  Jr.,  402  SMXG 

•  "Its  how  big?  How  will  you  deploy  it  without  killing  my  team  and  my  program?",  Mr.  William  Borkowski,  Jr.,  Raytheon  Missile  Systems 


Track  2:  Practical  Guidance 

•  Are  You  Making  the  Most  of  Your  Project  Schedules?,  Ms.  Susan  Byrnes,  PMP,  Natural  SPI,  Inc. 
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•  Keeping  the  Team  Motivated  for  Success  and  “Barrier  Busting”  -  Obtaining  Active  Leadership  Support,  Mr.  Michael  D.  Scott,  Raytheon  Missile  Systems 

•  Using  a  Level  3  Process  to  Achieve  CMMI  Level  3,  Mr.  Stephen  Ross,  Raytheon  Company 

•  Accelerating  Process  Improvement  through  Collaboratio:  The  NAVAIR  Systems  Process  Improvement  Community  of  Practice,  Ms.  Katie  Smith,  Naval  Air 
Systems  Command 

•  What  the  CMMI  Doesn't  Say  About  Training  (But  Should!),  Sree  Yellayi,  Siemens  Corporate  Research 

•  CMMI  CP  2.8  Interpretation  and  Implementation:  Is  This  Practice  Just  About  Numbers?,  Mr.  Lester  Stamnas,  Norausky  Process  Solutions,  Inc. 

•  Creating  Helpful  process  Directives,  Mr.  Kenneth  I.  Weinberg,  Raytheon  Company 


Track  3:  Appraisals 

•  Lessions  Learned  in  Helping  Large  and  Small  Organizations  Prepare  for  their  First  Appraisal,  Mr.  Robert  J.  Pomietto,  Center  for  Systems  Management 

•  Behind  Closed  Doors,  Mr.  Tom  G.  Lienhard,  Raytheon  Missile  Systems 

•  CMMI  Appraisal  Results:  The  Shocking  Truth  Revealed  and  Lead  Appraisers  Gone  Wild,  Ms.  Margaret  A.  Glover,  SEI 

•  Improving  Document  Reviews  for  Appraisals,  Mr.  Kent  McClurg,  Raytheon  Company 

•  Finding  CMMI  Compliant  Artifacts  and  a  Needle  in  a  Haystack,  Adrio  J.  DeCicco,  Raytheon  Company 

•  Lessons  Learned  and  Best  Practices  for  Evidence  Collection  in  Preparation  for  a  SCAMPI  Appraisal,  Mr.  Ben  Berauer,  Raytheon  Company 

•  Maximizing  Value  for  SCAMPI(SM)  Preparation,  Ms.  Joan  Weszka,  Lockheed  Martin  Corporation 


Track  4:  ROI  &  Benefits  of  CMMI 

•  Evaluating  the  Impact  New  Tools  and  Technologies  Using  Simulation,  Dr.  David  M.  Raffo,  Portland  State  University 

•  The  ROI  Dashbord  (c):  Understanding  the  Benefits  of  CMMI,  Mr.  Thomas  L.  McGibbon,  ITT  Industries,  AES 

•  Quality  Assurance  Involvement  Compared  to  Program  Results,  Ms.  Jill  Brooks,  Raytheon  Company 

•  Rapidly  Achieving  Measurable  ROI  Using  Early  Defect  Detection,  Mr.  Timothy  G.  Olson,  Quality  Improvement  Consultants,  Inc 

•  CMMI  Process  Improvement:  Its  not  a  technical  problem,  its  a  people  problem,  Mr.  Rolf  W.  Reitzig,  Cognence,  Inc. 

•  A  Project's  Perspective  of  a  CMMI  Level  5,  Mr.  Warren  Scheinin,  Northrop  Grumman  Corporation 

•  Achieving  the  Promised  Benefits  of  CMMI,  Dr.  Rick  Hefner,  Norhtrop  Grumman  Corporation 

•  Measuring  Economic  Benefits  of  Process  Improvement  in  CMMI  Level  1  Organization,  Dr.  Aldo  Dagnino,  ABB,  Inc.,  US  Corporate  Research 


Track  5:  High  Maturity 

•  Logarithms  Can  Be  Your  Friends:  Controlling  Peer  Review  Cost?,  Dr.  Richard  L.  W.  Welch,  Northrop  Grumman  Corporation 

•  Journeys  on  the  Road  to  Level  5,  Mr.  Joseph  V.  Vanderville,  Northrop  Grumman  Corporation 

•  Lessons  Learned  on  the  SCAMPI  Road  to  CMMI-Software  Level  5,  Mr.  Joseph  N.  Frisina,  BAE  Systems 

•  Merging  Measurement  in  Mature  Companies  -  A  Success  Story  of  Measurement  Process  Integration,  Ms.  Sharon  Rohde,  Lockheed  Martin  IS&S 

•  The  Road  to  Process  Improvement  Successes:  CMMI  Level  5/ISO  9001-2000  Business  Model,  Mrs.  Debra  S.  Roy,  BAE  Systems,  National  Security  Solutions 

•  Reducing  Variation  at  Each  CMMI  Maturity  Level?,  Mr.  Timothy  Kasse,  Kasse  Initiatives,  LLC 

•  Ways  to  Ensure  the  Culture  Supports  Level  5,  Mr.  Warren  Scheinin,  Northrop  Grumman  Corporation 

•  Analyzing  Defects  Can  Tell  a  Story  About  a  Company?,  Ms.  Diane  A.  Mitzukami-Williams,  Northrop  Grumman  Corporation  Mission  Systems 


Track  6:  Transition  to  CMMI 

•  Combining  Six  IPTS  and  Transitioning  to  CMMI,  Ms.  Judy  Overhouser-Duett,  NAVAIR 

•  How  to  Transition  Models  and  Disciplines  -  Looking  for  Transition  in  all  the  Wrong  Places,  Ms.  Lori  G.  Smailes,  TYBRIN  Corporation 

•  Using  SW-SMM  SQA  Independent  Verification  as  a  First  Step  for  the  Transition  to  CMMI,  Mr.  Alfredo  N.  Tsukumo,  CenPRA-Centro  de  Pesquisas  Renato 
Archer 

•  Service  Extensions  to  the  CMMI,  Mr.  Craig  R.  Hollenbach,  Northrop  Grumman  Corporation 

•  Applying  CMMI  to  Services,  Mr.  Juan  C.  Ceva,  Raytheon  ITSS 

•  Management  Challenges  &  Lessons  Learned  Implementing  CMMI  in  a  Services  Environment,  Mr.  Thomas  E.  Zience,  BAE  Systems  Information  Technology 

•  CMMI  vl.l  for  a  Service  Oriented  Organization,  Mr.  Steven  K.  Hall,  Raytheon  Corporation 


Track  7:  Measurement 

•  Software  Size  Growth  and  Uncertainity  -  Both  Affect  Estimate  Quality  and  Project  Risk  Presentation  and  Paper,  Mr.  Michael  A.  Ross,  Galorath,  Inc. 

•  Building  an  Automated  System  to  Support  Measurement  in  CMMI,  Dr.  Richard  Hayden,  Pragma  Systems  Corporation 

•  Team  of  Three  -  How  to  Get  Program,  Functional  and  Process  Management  Working  Together,  Mr.  Mark  A.  Marsh,  Raytheon  Company 

•  Parametric  Project  Monitoring  and  Control:  Performance-Based  Progress  Assessment  and  Prediction  Presentation  and  Paper,  Mr.  Michael  A.  Ross,  Galorath, 
Inc. 

•  Measuring  and  Estimating  Process  Performance,  Dr.  Richard  D.  Stutzke,  SAIC 

Thursday.  17  November  2005 


Technical  Sessions 


Track  1 :  CMMI  Process  Improvement 

•  “Barrier  Busting”  -  Obtaining  Active  Leadership  Support,  Mr.  Michael  D.  Scott,  Raytheon 

•  Don’t  Write  the  Wrong  Processes!,  Ms.  Suzanne  B.  Zampella,  The  Center  for  Systems  Management 
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•  Contrasting  CMMI  Contrasting  CMMI  and  the  PMBOK,  Mr.  Wayne  Sherer,  Anteon  Corporation 

•  Being  Customer  Oriented,  Mr.  Tim  Kasse,  Kasse  Initiatives,  LLC 

•  Learning  from  Lessons  Observed  -  Mitigating  Resistance  to  Process  Improvement,  Mr.  Bob  Norris,  National  Geospatial-Intelligence  Agency 


Track  2:  Practical  Guidance 

•  Supplier  Management  Strategy  Considerations  with  CMMI,  Mr.  Rick  Hefner,  Northrop  Grumman  Corporation 

•  Simplifying  Process  Tailoring  To  Enhance  Project  Execution,  Mr.  Howard  T.  Kaplan,  Raytheon  Company 

•  CMMI  and  agile:  a  High  Tech  R&D  Success  Story,  Mr.  Gene  Miluk,  SEI 

•  How  to  Incorporate  “Lessons  Learned”  for  Sustained  Process  Improvements,  Mr.  Anil  K.  Midha,  BAE  Systems 

•  Data  Management:  The  Hidden  Enabler  or  (The  Key  Data  and  Work  Product  Integrator),  Mr.  Lester  Stamnas,  Norausky  Process  Solutions 


Track  3:  Appraisals 

•  Techniques  for  Shortening  the  Time  and  Cost  of  CMMI  Appraisals,  Mr.  Sam  Fogle,  Systems  and  Software  Consortium,  Inc. 

•  Using  Classified  Programs  in  CMMI  Appraisals,  Mr.  Kenneth  I.  Weinberg,  Raytheon  Company 

•  The  Best  Intentions  of  SCAMPI  VI .  1 :  What  We  Meant  and  What  Some  People  Heard,  Mr.  Will  Hayes,  SEI 

•  A  Quantitative  Comparison  of  SCAMPI  A,  B,  and  C,  Mr.  Dan  Luttrell,  Northrop  Grumman  Mission  Systems 

•  Performing  Consistent  Appraisals  in  a  Global  Organization,  Ms.  Jeanine  Courtney-Clark,  Integrated  System  Diagnostics,  Inc 


Track  4:  ROI  &  Benefits  of  CMM  I  /  SCAMPI  B/C 

•  The  Effects  of  CMMI  on  Program  Performance,  Mr.  Joseph  P.  Elm,  SEI 

•  Squeezing  Variation  for  Profit,  Mr.  Donald  R.  Corpron,  Northrop  Grumman  Corporation 

•  Process  In  Execution  Review  (PIER)  and  the  SCAMPI  B  Method,  Ms.  Lorraine  J.  Adams,  SEI 

•  Planning  a  SCAMPI  C  Appraisal  from  a  Strategic  Perspective,  Mr.  John  P.  Kennedy,  The  Mitre  Corporation 

•  Critical  Path  SCAMPISH  Getting  Real  Business  Results  from  Appraisals,  Mr.  Michael  J.  West,  Natural  SPI,  Inc. 

•  Using  SCAMPI  C  for  Collective  Improvement  Across  a  Multi-Business  Program,  Mr.  Oktawian  Nowak,  Motorola,  Inc. 


Track  5:  High  Maturity 

•  A  Statistical  Approach  to  Product  Quality  Assurance,  mr.  Randall  J.  Varga,  BAE  Systems 

•  The  Key  to  a  High  Maturity  Rating  is  -  ORGANIZATION,  Mrs.  Karen  M.  Pelletier,  Northrop  Grumman  Corporation 

•  Paladin  Drives  Forward  To  CMMI®  Maturity  Level  5,  Mr.  Victor  Elias,  M.S.,  Armament  Software  Engineering  Center,  US  Army 

•  Business  Improvements  Achieving  CMMI®Level  5  at  SAIC:  Who  Keeps  Moving  My  Process?,  Ms.  Sharon  Cobb  Flanagan,  SAIC 

•  Extending  CMMI  Level  4/5  Organization  Metrics  Beyond  Software  Development,  Ms.  Linda  R.  Brooks,  Northrop  Grumman  Corporation 


Track  6:  CMMI  Extensions 

•  Capability  Maturity  Model  Integration  (CMMI®)  Tailoring  for  an  IT/MS  Services  Environment,  Ms.  Stacy  Savage,  BAE  Systems  Information  Technology 

•  Adapting  CMMI  for  Acquisition  Organizations:  A  Preliminary  Report,  Dr.  Hubert  Hofmann,  General  Motors 

•  How  to  Become  Your  Customer’s  Software  Provider  of  Choice,  Mr.  David  Herron,  DCG,  Inc. 

•  Space  and  Missile  Systems  CenteSpace  and  Missile  Systems  Center,  Mr.  Keith  Wright,  SPARTA,  Inc. 

•  Software  Outsourcing  with  CMMI,  Dr.  John  W.  Mishler,  SEI 


Track  7:  Systems  Engineering 

•  Sound  Systems  Engineering  using  CMMI®,  Ms.  Sandee  D.  Guidry,  TECHSOFT 

•  Systems  Engineering  Influence  Throughout  the  CMMI,  Mr.  Tim  Kasse,  Kasse  Initiatives,  LLC 

•  Future  of  System  and  Software  Engineering  Project  Management  and  the  CMMI,  Dr.  Kenneth  E.  Nidiffer,  Systems  and  Software  Consortium 
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CMMI  is  registered  in  the  US  Patent  &  Trademark  Office  by  Carnegie  Mellon  University 


Sunday,  November  13,  2005 


12:00  PM  -4:00  PM 

Registration  for  Conference  and  Tutorial 

Atrium 

Monday,  November  14,  2005 


7:00  AM  -5:00  PM 

Tutorial  Registration  ($200  Tutorial  Fee) 

Atrium 

7:00  AM  -8:00  AM 

Continental  Breakfast  (Tutorial  Attendees  Only) 

Atrium 

8:00  AM  -5:00  PM 

CMMI  Tutorial  Tracks  (Tutorial  Attendees  Only) 

See  Following  Pages 

12:00  PM  - 1:00  PM 

Lunch  (Tutorial  Attendees  Only) 

Grand  Mesa  ABC 

5:00  PM  -  6:30  PM 

Reception  (All  CMMI  Conference  Attendees) 

Display  Area 

Tuesday,  November  15,  2005 


7:30  AM  -8:30  AM 

Registration  and  Continental  Breakfast 

Atrium 

8:30  AM  -8:45  AM 

Opening  Remarks 

Grand  Mesa  DEF 

8:45  AM  -9:30  AM 

Grand  Mesa  DEF 

Session  A 

LTG  Joseph  Yakovac,  USA,  Military  Deputy,  Office  of  the  Secretary  of  the  Army, 
Acquisition,  Logistics  &  Technology 


9:30  AM  - 10:00  AM 

Break 

Atrium 

10:00  AM  -  12:00  PM 

Session  B 

Executive  Panel  -  “How  Has  CMMI  Improved  Our  Program  &  Project 
Performance  -  Or  Has  it?” 

Moderator: 

Mr.  Mark  Schaeffer,  Director,  Systems  Engineering,  OUSD(AT&L) 

Defense  Systems  and  OSD  Sponsor,  CMMI 

Grand  Mesa  DEF 

Panelists: 

Mr.  Dev  Banerjee,  Division  Director,  Systems  &  Flight  Engineering,  Boeing 
Integrated  Defense  Systems 

Mr.  John  Evers,  Raytheon  Processes  Program  Manager,  Raytheon  Common 

Engineering  Process  Program 

Brig  Gen  Gary  Salisbury,  USAF  (Ret),  Executive  Director,  Business  Development, 
Defense  Mission  Systems,  Northrop  Grumman  Mission  Systems 


CONFERENCE  AGENDA 


CONFERENCE  AGENDA 


12:00  PM  -  1:30  PM 

Lunch 

CMMI  -  State  of  the  Model 

Mr.  Bob  Rassa,  Raytheon;  Mr.  Clyde  Chittister,  SEI 

Grand  Mesa  ABC 

1:30  PM  -5:00  PM 

Technical  Sessions 

See  Following  Pages 

3:00  PM  -  3:30  PM 

Break 

Display  Area 

5:00  PM  -  6:30  PM 

Reception 

Display  Area 

Wednesday,  November  16,  2005 


7:00  AM -8:00  AM 

Registration  and  Continental  Breakfast 

Atrium 

8:00  AM -5:00  PM 

Technical  Sessions 

See  Following  Pages 

9:30  AM  -  10:30  AM 

Break 

Display  Area 

12:00  PM  -  1:30  PM 

Lunch 

Conference  Best  Paper  Awards 

Grand  Mesa  ABC 

3:00  PM  -  3:30  PM 

Break 

Display  Area 

Thursday,  November  17,  2005 


7:00  AM -8:00  AM 

Registration  and  Continental  Breakfast 

Atrium 

8:00  AM  -2:30  PM 

Technical  Sessions 

See  Following  Pages 

9:30  AM  -  10:30  AM 

Break 

Display  Area 
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Presentation  Outline 


•  Methodologies  and  approaches 

•  Lessons  learned 

•  Best  practices  for  appraisal 
preparation  and  evidence 
collection 

•  Summary  take-aways 


^v/juxRm] 


Methodologies  and  Approaches  Taken 

•  Program  scorecard  based  on  Objective  Evidence  (OE) 

-  Collecting  and  documenting  OE  follows  a  disciplined  data  collection 
and  scorecarding  process 

-  Customizing  the  appraisal  tool  to  meet  the  collection  process 

-  Pllb  building  using  appraisal  tool  with  direct  linkage  into 
organizational  Process  Asset  Library 

•  Evidence  verification 

-  Collecting  the  right  direct  and  indirect  evidence 

-  Focusing  on  the  required  (expected)  evidence  ...  don't  try  to  inundate 
with  unessential  data  or  “almost"  the  right  thing 

-  Identifying  evidence  using  OE  Collectors,  FARs,  Verifiers 

•  Gap  analysis  and  closure 

-  Detailing  action  plans  targeting  identified  deficiencies 

-  Collecting  OE  until  specif  ied  scoring  criteria  are  met 
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Pre- Appraisal  Scorecarding 
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Methodologies  and  Approaches  Taken  -  2 

•  Evidence  collectors 

-  Populate  appraisal  tool  with  appropriate  direct  and  indirect  OE 

-  Tag  data  when  linked  to  a  practice 

•  Evidence  verifiers 

-  Review  each  practice  for  adequate  evidence  based  on  program 
scope,  discipline  responsibilities,  etc. 

-  Tag  data  to  indicate  verification  results 

-  Mentor  evidence  collectors 

•  Class  C  and  B  appraisals  validate  that  right  evidence  was 
provided 

-  Tag  data  to  indicate  practice  implemented  and  evidence  is 
satisfactory 

•  Loop  through  above  steps  as  needed  until  the  right 
evidence  is  captured 

-  Tagging  at  each  step  of  process  ensures  closure  on  any  evidence 
issues 
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Evidence  Tracking/Tagging"  Flow 


Evidence  is  initially  tagged  by 
Collectors  as: 

•Added- Direct/ Added-Indirect  - 
Evidence  that  has  been  added  as  a 
result  of  an  earlier  Class  B 

•Collector- Direct/Indirect  - 
Evidence  for  practices  that  were 
not  included  in  an  earlier  Class  B 


Evidence  is  reviewed  by  Verifiers  and 
tagged  as: 

•Ver-Direct/Indirect  -  Evidence  is  ready 
for  an  appraisal  and  requires  no  additional 
work.  * 

•Ver-Direct/Indirect- Rework  -  Evidence 
supports  the  practice  but  is  not  ready  for 
an  appraisal  and  needs  additional  work  by 
the  collector 

•Rejected-Remove  -  Evidence  does  not 
support  the  practice  and  should  be 
removed  by  the  collector 


Ready  for 
next 
appraisal 


All  Evidence 
Ver- Direct 
/Indirect?  * 


Collectors  re¬ 

work  evidence 
and  tag  it  as 
Collector 
Direct/Indirect 
Reworked 


Evidence  that  has  an  “Evidence  Type"  flag  of  Direct(A)  or  Indirect(B)  was 
accepted  in  an  earlier  Class  B.  For  this  effort,  this  evidence  is  considered 
verified  and  will  not  be  reviewed  by  the  verifiers. 


Verifiers  re- 

review  and  re¬ 

tag  evidence 
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"Evidence  Type"  Tagging 


For  each 
practice  in  the 

PA  ... 


MG09)  Element:  VER  SP  1.1 


liltering  Record 


m 


;da 


■mos  P2 


2 


Organization 


Good  D+l 


Good  D+l 


Rating  Level: 

|  Raytheon  (St  Pete) 


Good  D+l 


Good  D+l 


Not  Applicable 


lANIZATION 


Practr 


VER  SP1.1 
VER  SP  1.2 
VER  SP  1.3 
VER  SP  2.1 
VER  SP  2.2 
VER  SP  2.3 
VER  SP  3.1 
VER  SP  3.2 
VER  CO  1  (GP  2.1) 
VER  AB  1  (GP  3.1) 


SP1.1  Select 
Work  Products  for 
Verification 

Select  the  work 
products  to  be  verified 
and  the  verification 
methods  that  will  be 
used  for  each. 

Work  products  are 
selected  based  on 
their  contribution  to 
meeting  project 
objectives  and 
requirements,  and  to 
addressing  project 
risks. 

The  work  products  to 


Element  Records  j  Element  Documents  | 

+  New  Rec  IE§  Save  Rec  ^  Spell 


Delete  Rec  %  Cancel 


Rec  ID  ♦ 

Record  Type 

Status 

Verification 

Records  [7] 

3082 

DD(X) 

□  E  Examined 

Yes 

How  do  you  establish 

► 

3254 

CECDA 

□  E  Examined 

How  do  you  establish 

342S 

GAM  OS  P2 

□  E  Examined 

Yes 

How  do  you  establish 

3598 

E4B2 

□  E  Examined 

Yes 

How  do  you  establish 

3989 

CMMI  Std  Pll 

□  E  Requested 

How  do  you  establish 

4207  ST.  PETE  0E 

Direct 

4383  Evidence  Issr 

.  0E  Examined 

ACJ  8/9/2004  -  CEC 

Record  Fields  /  Projects  j  Elements  /  Data  Sources  /  T earn  Members  Record  Documents 
&  Open  Detach  ^  Add  /  Connect  Edit  Comments 


Verifier  reviews  the  evidence  and  rates  it: 

•Ver- Direct/Indirect  -  Evidence  is  ready  for 
an  appraisal  and  requires  no  additional  work. 

•Ver-Direct/Indirect-Rework  -  Evidence 
supports  the  practice  but  is  not  ready  for 
an  appraisal  and  needs  additional  work  by 
the  collector 

•Rejected-Remove  -  Evidence  does  not 
support  the  practice  and  should  be  removed 
by  the  collector 

•Contested  -  Program  position  on  practice 
may  not  be  acceptable  to  an 


Title 

Doc  ID 

Doc-Rec  Comments 

File  Name  or  URL 

Evidence 

^wr 

JSN  SSPM  -  Integrated  Test  Process 

7392764-301 

The  Scope,  pg.  1  of  this  docun 

http:  //businessweb.  stp.  us 

Ver-Direct  ^ 

CEC 

CEC 

Computer  Program  T  est  (CPT) 

7392764 

Section  3. 1.2.1,  pg.  4  describe 

http:  /  /  eds- web.  stp.  us.  ray. 

Ver-I  ndirect 

St  Petersburg  Systems  Engineering  Knowledge  Reposito 
Verification  Process  Description 

G3-0031 0-001 

EP4041 2.01 

SEKR  2-4  TEMP; 

http:  //www.  stp.  us.  ray.  cor 

Ver-I  ndirect 

0R( 

5.1  Prepare  for  Verification,  T  c 

http:  / /www.  stp.  us.  ray.  o  V er-l  ndirect 

Verification  Process  Directive 

EP40412 

5.1.1  S  elect  work  products  for 

http:  //www.  stp.  us.  ray.  o  V er-l  ndirect 

CEC 

CEC 

CEC 

SP  1.1  Indirect  -  LCCB  Meeting  Minutes  for  DDS 

8192 

ACJ  8/1 0/2004 -This  evidenc 

http:  //www-cf  1 .  stp.  us.  ray  Collector-1  ndirect 

CEC  Software  T est  Description  (STD)  -  Mode  and  State 

11896 

Reference  entire  document  as 

http:  /  /  eds- web.  stp.  us.  ray. 

http:  //eds- web.  stp.  us.  ray. 

Ver-Direct 

System/Software  Test  Plan  (STP)  for  CEC  Baseline  2.1 

11895 

See  Section  1.1  Identification, 

Ver-Direct 

page  1,  and  Section  1.4 
Document  Overview,  page  2-3 

an  appraiser  A 
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Evidence  Issues  Process 


Verifier  documents  issues 

with  the  evidence  for  a 
project  -  they  document  it 
in  an  evidence  record  for 
that  project. 


Rating  Level: 

|  Raytheon  (St  Pete) 


H. 


i.e. 


cable 


EI-DO(X) 

EI-CEC 

EI-GAMOS 

EI-E4B2 


t  Documents  | 

|  Rec  5$^  Spell 


■  Delete  Rec  Z  Cancel  Cha 


be  Status 


| Verification |  Records  [7] 


□  E  Examined 


Yes 


VER5P2.2  % 

VER  5P2.3 

— 

VER  SP  3.1 

— 

VER  5P3.2 

— 

VER  CO  1  (GP  2.1) 

— 

VER  AB  1  (GP  3.1) 

V  - 

□  E  Examined 
^MOS  P2  OE  Examined 
4B2  OE  Examined 


kMI  Std  Pll 

4207  SnSpETEOE 
4303  Evidence  Issr  OE  Examined 


How  do  you  establish  anc 
How  do  you  establish  anc 
How  do  you  establish  anc 
How  do  you  establish  anc 
How  do  you  establish  anc 
Direct 

I ACJ  3/3/2004 -CEC  DA 


Record  Fields  /  F 


ISPI.1  Select 


a 


Once  the  issues 
are  documented 
the  Verifier  sets 
the  status  to 
‘Open" 

nel^onmoucion  to 
I  meeting  project 
I  objectives  and 
I  requirements,  and  to 
(addressing  project 
|  risks. 

| The  work  products  to 


[  Data  Sources  /  Tealv 
f  Add /Connect  EdiY 


Doc  ID 


|3SN  SSPM 
Computer 
St  Pet. 
Verific 


Integ 

Progr^ 


Once  the  issues  have 
been  addressed  the 
Collector  changes 

the  status  to 
"Addressed" 


73327G4-301 
73327G4 
1-00310-001 
□41 2.01 
P412 


5. 

5.1 
ACJ 
Referi 
See  Seel 
page  1 , 
Document 


The  Verifier  then  reviews  the  collectors 
response  and  changes  the  status  to: 

•Additional_Data  -  Additional  evidence 
or  explanation  is  required  and  the 
collector  has  additional  work  to  do. 
When  the  work  is  complete  the 
collector  changes  the  status  back  to 
“Addressed"  the  process  loops  back  to 
the  verifier. 

•Rejected  -  The  verifier  disagrees  with 
the  evidence  or  explanation  provided 
and  the  collector  needs  to  resolve  with 
the  verifier.  When  the  work  is 
complete  the  collector  changes  the 
status  back  to  "Addressed"  the  process 
loops  back  to  the  verifier. 

•Closed  -  The  verifier  agrees  with  the 
evidence  or  explanation  provided  and  no 
additional  work  is  required. 


Overffl 
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Verification  Process  (cont.) 


I  Options  Filtering  Element  Filtering  Record 
iModel:  |CS11 


ORGANIZATION 

Practice 

A 

VER  SP1.1 

VER  SP  1.2 

VER  SP  1.3 

■  I 

H 

VER  SP  2.1 

VER  SP  2.2 

VER  SP  2.3 

> 

VER  SP  3.1 

I 

VER  SP  3.2 

VER  00  1  (GP  2.1) 

VER  AB  1  (GP  3.1) 

V 

SP3.1  Perform 
Verification 

A 

Derform  verification  on 
:he  selected  work 
products. 

Verifying  products  and 
work  products 
ncrementally 
promotes  early 
detection  of  problems 

J 

Record  Fields  /  Projects  |  Elements  /  Data  Sources  /  T earn  Me 


&  Open  Detach 


Add  /  Connect 


'1  Ed 


Title 


Doc 


St  Petersburg  Systems  Engineering  Knowledge  Reposito  63-C 
Verification  Process  Description  ,/■■— v  EP4 

Verification  Process  Directive 


Once  all  of  the  PAs  have  been 
rated  green  there  is 
concurrence  between  the  OE 
Collectors  and  Verifiers  that 
the  practice  is  adequately 
supported  and  ready  for  the 
next  appraisal 


EP4 

aseline2.1  118! 
e  and  State  1 1 8S 
7560\ 
748643 
7486495-l 
7486485-02? 
8294 


Based  on  the  evidence  for  each  practice. 

Verifier  scorecards  each  project  as: 

Good_D+I  -  All  direct  and  indirect  evidence  is  ready  for 
the  appraisal  and  requires  no  additional  work 

•  Insufficient_Direct  -  Direct  evidence  is  not  ready  for 
an  appraisal  and  requires  additional  work  by  collectors, 
but  the  indirect  is  ready  and  requires  no  additional  work 

•Insufficient_Indirect  -  Indirect  evidence  is  not  ready 
for  an  appraisal  and  requires  additional  work  by 
collectors,  but  the  direct  is  ready  and  requires  no 
additional  work 

•Insufficient_D+I  -  Offered  evidence  has  been  reviewed, 
but  both  the  direct  and  indirect  are  not  ready  for  an 
appraisal  and  requires  additional  work  by  collectors 

•Evaluation_not_complete  -  Either  no  evidence  has  been 
reviewed  or  not  all  of  the  evidence  has  been  reviewed 
and  requires  additional  work  by  the  verifier 
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Program  B  Stoplight  Status 
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2.4 
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PT 
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1.4 

1.5 
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1  2|  1  3|  1.4 

CD 
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Legend/Count 
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Raytheon 

Lessons  Learned 


•  Appraisal  preparation  requires  tooling 

-  Flexible  appraisal  tools  supporting  preparation  are  very  important 

-  Tool  must  be  flexible  and  conf igurable 

•  Use  the  same  tools  for  appraisal  preparation  and  the  appraisal 

-  Scorecard  readiness  using  the  appraisal  tool 

-  Using  the  tool  as  a  window  to  the  organization's  PAL  (not  a  separate 
collection  of  evidence) 


•  Tools  are  not  enough 

-  Need  to  have  scorecarding  requirements  and  features  defined 

-  Need  a  well  thought  out  scorecarding  process  that  is  both  implemented  and 
followed 


-  Appraisal  tools  did  not  adequately  support  appraisal  preparation  right  out 
of  the  box 


•  Every  tool  has  it's  bugs  and  hidden  “features" 

-  Need  tool  "wizard"  to  ensure  features  are  implemented, 
and  ensure  any  tool  problems  do  not  affect  progress 


November  14-17,  2005  Page  11 


Lessons  Learned  (cont.) 

•  You  may  not  always  have  the  right  people  collecting  data 

-  Collectors  of  program  OE  must  have  program  data  repository  and 
work  product  knowledge 

-  FARs  must  be  the  ones  that  do  the  work  and  are  familiar  with  how 
they  do  it  and  what  they  produce 

-  Evidence  verifiers  must  be  familiar  with  needed  OE 

-  What  you  see  is  what  you  get  ...  OE  collected  must  support  FAR 
story  (This  connection  is  KEY  to  the  success  of  the  appraisals) 

-  Evidence  collectors  may  not  be  FARs  !!?? 

■  FARs  are  typically  key  program  personnel 

■  Programs  are  resistant  to  dedicate  key  program  personnel  to  OE 
collection 

■  FARS  must  see  /  understand  collected  evidence 


0 
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Best  Practices  - 
Evidence  Collection  (1) 


•  Use  PIID  questions  to  guide  the 
process 

-  Guides  the  collection  team  to  what  needs  to 
be  collected  for  a  given  program 

-  Shows  compliance  with  the  org  processes  by 
answering  the  question  for  your  program, 
for  each  practice, 

-  Provides  discipline  and/or  support  function 
specific  unique  answers,  if  applicable 

-  Explains  any  life-cycle  or  other  program 
considerations  that  affect  how  the  practice 
is  implemented,  and  the  evidence  to  support 
them 

-  Weaves  the  story  of  how  it  is  done,  and 
what  work  products  are  produced,  and  then 
provide  those  work  products  as  evidence 
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Best  Practices  -  Rr  moor 

Evidence  Collection  (2) 

•  Focusing  on  the  principle  "direct  evidence",  the  rest  will 
come 

-  Started  with  both  direct  and  indirect  evidence  collection  direction 

-  Found  the  indirect  evidence  usually  came  naturally 

•  Focusing  on  providing  the  major  program  work  products  as 
evidence  everywhere  they  applied 

-  SOP,  SEMP,  PMP,  IMP/IMS,  etc. 

•  Building  evidence  threads  across  practices  and  even 
process  areas 

-  Especially  for  the  GPs 

•  Look  for  consistency  with  organization  procedures 

-  Keep  a  cross-program  focus  for  consistency  and  common  evidence 
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Best  Practices  - 
Evidence  Collection  (3) 


A  close  working  relationship  between 
the  program's  FAR,  the  Verifier,  and 
the  evidence  collectors 

-  Evidence  Collectors  and  FARs  provide 
program  expertise  in  work  products 
produced 

-  Verifiers  provide  CMMI  model/method. 
Organizational  Process  expertise,  and 
evidence  coordination 

-  OE  supports  what  the  FARs  describe  as 
standard  practices,  and  the  model! 

-  Team  review  of  expected  work  products  for 
each  model  practice 


November  14-17,  2005  Page  15 


Best  Practices  - 
Evidence  Review 


_3Lt\/I  INI 


•  Reviewing  evidence  across  programs  to  ensure  consistency 

-  Understand  the  organizational  standard  process,  and  focus  on 
common  program  responses,  explaining  any  tailoring  or  program 
unique  behaviors 

-  Identifying  and  ensuring  all  programs  had  similar  "right"  data 

•  Identify  where  evidence  does  not  exist,  and  needs  to  be 
produced  !!!!!! 

-  shouldn't  be  too  many  cases  of  non-existent  data 

•  Review  regularly  and  provide  corrective  action  feedback 
promptly 

-  Drive  the  evidence  collection  to  completion,  and  get  the  right  stuff! 


A 


'Jr 


November  14-17,  2005  Page  16 


Best  Practices  -  ^ 

Preparation  Monitoring  and  Control 

•  Monitor  and  report  status  of  every  practice 

•  Review  with  appropriate  management  drives  the  process 

-  This  can  be  both  a  positive  and  negative  driver 

•  Know  your  status  at  all  times 

•  Maintain  action  item  and  action  plan  status 

-  Ensure  that  all  'to  do's"  get  done  promptly 

-  Plan  appropriate  correction  actions  plans  to  address  issues 

-  Set  due  dates  that  achieve  the  desired  result 

-  Identify  and  track  risks,  and  develop  risk  mitigation  plans 

•  Collect  OE  until  you  meet  specific  scoring  criteria 

-  Iterate  process  until  ready  for  appraisal 
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Raytheon 

In  Summary 

•  Collecting  and  documenting  OE  requires  a  well  defined  and  disciplined 
process,  just  like  the  appraisal 

•  Objective  Evidence  PIID's  are  central  in  how  we  prepare  for  the 
appraisals 

•  Appropriate  tools  can  greatly  facilitate  preparation 

-  Using  the  same  tools  for  preparation  and  the  appraisal  is  a  big  plus 

•  Determining  if  a  project's  OE  is  appropriate  and  adequate  is 
ultimately  left  up  to  CMMI  appraisers 

-  But  developing  appropriate  OE  database  is  key  to  preparing  for  the 
appraisal 
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Supplemental  Charts 


Section  Divider 


Some  Terms  Used 


Appraisal  Scorecard:  A  scorecard  showing  how 
well  prepared  for  an  appraisal  a  program  is. 
Can  be  OE  focused  (bo  we  think  we  have  the 
right  evidence). 

Scorecarding:  The  procedural  steps  followed  to 
collect,  validate,  monitor,  and  control 
preparations  using  a  scorecard. 

PIIDs:  Practice  Implementation  Indicator 

Database  showing  what  OE  your  organization 
and  programs  expect  for  each  practice  of  the 
CMMI  Model,  and  what  each  program  has  to 
meet  that  expectation. 
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Using  CMMI  to  Dig  Out  from 
Ad  Hoc  Practices 

10-15-05 
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Overview 

•  SW  Project  Overview 

-  Deployed  Quickly 

-  Not  enough  testing 

-  A  lot  of  unhappy  customers 

•  Result 

-  20  SW  Splinters 

-  Overwhelmed  with  maintenance 

-  Created  more  problems 
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Recovery  Plan 

•  It  was  clear  there  was  a  problem 

-  Gain  support  to  use  process 

•  Define  Processes  and  Procedures  to 
Recover 

-  SW  Helpdesk 

•  Train  the  Organization  on  the  new 
processes 
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Recovery  Plan 

•  Establish  a  method  to  review,  prioritize 
and  approve  SW  issues 

-  Helpdesk  Review  Team 

•  Make  sure  each  Issue  is  clearly  defined 

-  Consistent  bug  Reporting 

•  Plan  the  releases  for  approved  issues 

-  Assigned  a  Project  Manager 

•  Monitor  the  execution  to  the  Plan 
-Weekly  Meetings 
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Recovery  Plan 

•  Design  and  Implement  the  solutions 

-  Developed  a  streamlined  process 

•  Integrate  solutions  into  a  SW  Release 

-  SW  Group  Leader  coordinated  integration 

•  Verify  the  solution  works 

-Test  Report  Required 

•  Configuration  Management 

-  SW  Release  was  controlled  with  CM  SW 
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Recovery  Plan 

•  Validate  the  SW  Release 

-  SQA  Group 

•  Performed  audits  and  system  testing  on  CM 
Controlled  Release 

•  Performed  audits  of  the  process 

•  Conducted  Lessons  Learned 

•  Collected  metrics  monthly 
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Organization  Process  Focus 

•  SP1.1-1  Establish  Organization  Process 
Needs 

-  Better  methods  to  collect  SW  issues 

-  Review  the  issues  received 

-  Track  and  close  issues  being  worked  on 

-  Better  development  methods 
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Organization  Process  Focus 

•  SP1 .1-2  Appraise  Organization’  Processes 

-  Strengths 

•  Good  Skills.  Well  trained  on  the  development 
environment/tools 

•  Consistently  used  CM  system.  All  splinters  were 
controlled 

-  Weaknesses 

•  Bug  reporting  system  was  difficult  to  use 

•  Bug  reporting  system  was  not  being  used 
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Organization  Process  Focus 

•  SP1 .3-1  Identify  the  Organization  Process 
Improvements 

-  Web  based  SW  Helpdesk 

-  The  Helpdesk  Database  was  tied  to  the  CM  System 

-  Rotated  SW  Engineers  through  Helpdesk 

-  Instituted  a  Review  Team 

-  Developed  an  escalation  procedure 

-  Project  Manager  coordinated  customers  and  SW 
team 
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Organization  Process  Focus 

•  SP2.1-1  Establish  Process  Action  Plans 

•  SP2.2-1  Implement  Process  Action  Plans 

•  SP2.3-1  Deploy  Organizational  Process 
Assets 

-  SW  Helpdesk  Procedure 

-  SW  Escalation  Procedure 

-  SW  Planning  and  Numbering  Procedure 

-  SW  Installation  Procedure 


10 


NexSummit  LLC  908-684-8914  nexsummit.com 


Organization  Definition 

•  Establish  Standard  Processes 

-  SW  Helpdesk  Procedure 

•  Defined  Helpdesk  as  a  repository  not  an  escalation  path 

•  Defined  the  CM  system  as  the  repository.  All  issues  and 
revisions  were  controlled.  Also  had  a  web  interface. 

•  Defined  the  role  of  the  Helpdesk.  Verify  Information  is 
complete,  assign  CR  number. 

•  Defined  the  role  of  the  Expert.  Verify  that  the  report  was 
technically  complete 

•  Defined  the  role  of  the  Review  Team.  Set  priority  and  defined 
the  scope  of  work. 
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Organization  Definition 

•  Establish  Standard  Processes 

-  SW  Helpdesk  Procedure 

•  Defined  the  role  of  the  Project  Manager.  Plan  SW 
Releases  based  on  priorities  and  customer  needs. 

•  Defined  the  role  of  the  SW  Group  Leader.  Assign 
issues  to  developers  with  the  right  skill  set. 

•  Defined  the  methods  used  to  track  the  issue 
through  the  development  process. 

•  Defined  the  steps  to  enter  an  issue  via  the  web 
site. 
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Organization  Definition 


Establish  Standard  Processes 

-  SW  Escalation  Procedure 

•  Defined  what  an  escalation  is.  Safety  or  Severe 

•  Defined  a  single  point  contact 

•  Defined  the  methods  to  report  an  escalation 

_  0\ A/  pipnplt^a _ _ Klii P rOPPQQ 

Alpha  Releases  used  to 


Defined 

required 

splinters 


X.XXX  A1 


Beta  sw  to  a  specific  9me.  New  scheme 

customer  before  merging  3  while  meraino 
into  the  trunk  °  y  y 
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Organization  Definition 

•  Establish  Standard  Processes 

-  SW  Installation  Procedure 

•  Lessons  Learned  revealed  that  many  of  the  reported  issues 
were  due  to  installation  problems. 

-  Incorrect  Installation 

-  Couldn’t  back  out  the  software 

•  Defined  how  to  Back  up  Files 

•  Defined  How  to  Install  new  SW 

•  Defined  Trouble  Shooting  methods  and  work-arounds 

•  Defined  how  to  Back  out  the  SW 

•  Defined  metrics  to  track  successful  installations  and 
escalations 
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Organizational  Training 

•  SGI  Establish  Organization  Training 

•  SG2  Provide  Necessary  Training 

-Training  provided  to  the  Rotating  Helpdesk 
members,  the  Experts,  and  the  Developers 

-Training  on  the  website  interface  was 
provided  to  the  users  who  will  be  reporting 
issues  to  helpdesk 

-Training  records  were  maintained 
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Project  Planning 

•  SP1 .1-1  Establish  the  Scope  of  the  Project 

-The  Project  Manager... 

•  Used  the  severity  rating  assigned  by  the  Review 
Team  to  identify  the  which  issues  would  be 
addressed  in  each  release 

•  Communicated  the  plan  to  the  customers 

•  Negotiate  scope  if  necessary 
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Project  Planning 

•  SP1.2-1  Establish  Estimates 

-  The  Review  Team  used  three  categories  to 
size  the  effort 

•  Category  1 :  Bug  Fix 

•  Category  2:  Small  Feature 

•  Category  3:  Large  Feature 

-  The  Project  Manager  used  5,  20  and  40  days 
to  estimate  the  effort 

•  Avoids  interrupting  the  SW  Developers  for 
estimates 
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Project  Planning 

•  SP1 .3-1  Define  Project  Life  Cycle 

-  The  Helpdesk  Procedure  defined  the  Life  Cycle: 

•  Open  -  Initial  Setting 

•  In  Expert  Review  -  Helpdesk 

•  In  Issue  Review  -  Expert 

•  In  Planning  -  Review  Team 

•  In  Development  -  Project  Manager 

•  Ready  to  Merge  -  SW  Developer 

•  In  SQA  -  SW  Group  Leader 

•  Complete  -  SQA  Group  Leader 
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Project  Planning 

•  SG2  Develop  Project  Plan 

-  The  Project  Manager  is  responsible  for  compiling 
plan. 

-  Main  risk  was  that  the  category  selected  was 
incorrect.  Issues  were  re-estimated  when  assigned 
for  correction. 

-  All  work  products  were  linked  to  the  Helpdesk  entry. 

-  Resources  were  reviewed  on  a  monthly  basis 

-  Skill  sets  were  reviewed  monthly. 

-  Stakeholders  reviewed  the  plan  and  Project  Manager 
acquired  necessary  approvals. 
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Project  Monitor  and  Control 

•  SGI  Monitor  Project  Against  Plan 

-  Project  Manager  conducted  weekly  meetings 
to  monitor  progress. 

-  Problems  identified. 

•  Resource  limitations:  People  or  Equipment 

•  SG2  Manage  Corrective  Action  to  Closure 

-  Resources  were  addressed 

-  Customer  was  notified  in  advance  when  a 
delay  was  likely. 
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Requirements  Management 

•  SGI  Manage  Requirements 

-  All  software  issues  are  entered  into  helpdesk  with  a 
standardized  report  form. 

-  The  Review  Team  verified  the  information  was 
complete  and  accurate  before  committing. 

-  All  changes  were  made  through  Helpdesk  and  were 
under  CM  control. 

-  Project  Manager  tracked  that  each  issue  was 
assigned. 

-  SQA  tracked  that  each  issue  in  the  plan  was  included 
and  closed. 
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Requirements  Development 

•  SGI  Develop  Customer  Requirements 

-  Helpdesk  entries  were  used  as  the  customer 
requirements 

•  SG2  Develop  Product  Requirements 

-  SW  Requirements  were  developed  fro  CAT2  and  3 
projects 

•  SG3  Analyze  and  Validate  Requirements 

-  Analysis  was  performed  by  the  expert  prior  to 
submission  to  the  Review  Team. 

-  Review  Team  was  the  final  validation  step. 


22 


NexSummit  LLC  908-684-8914  nexsummit.com 


Technical  Solution 

•  SGI  Select  Product  Components  Solution 

-  Primarily  applied  to  Category  3  projects 

•  SG2  Develop  Design 

-  Designs  were  develop  for  Category  2  and  3 

•  Implement  the  Product  Design 

-  SW  code 

-  SW  Test  Results 

-  SW  Release  Notes 
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Product  Integration 

•  SGI  Prepare  for  Product  Integration 

-  The  SW  Group  Leader  developed  the  Integration 
Plan. 

-  The  Project  Manager  track  progress. 

•  SG2  Ensure  Interface  Compatibility 

-  The  SW  Group  Leader  was  responsible  for  reviewing 
SW  deliverables  and  interface  compatibility 

•  SG3  Assemble  Product  Components  and 
Deliver  Product 

-  The  SW  Group  Leader  coordinated  build  machines 
for  the  final  build  and  SW  delivery. 
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Verification 

•  SGI  Prepare  for  Verification 

-  The  SW  Engineers  are  required  to  develop  a  Test 
Plan  for  all  three  category  projects 

•  SG2  Perform  Peer  Reviews 

-  Peer  Reviews  are  conducted  for  Category  3 

-  SW  Group  Leaders  review  work  products  for 
Category  1  and  2  projects 

•  SG3  Verify  Selected  Work  Products 

-  Software  Engineers  are  responsible  for  verifying  work 
products  and  documenting  Test  Results 
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Validation 

•  The  SQA  is  responsible  for  validating  the 
SW  Release 

-  Validate  all  issues  promised  are  included  in 
the  release 

-Auditing  Test  Results 

-  Validating  general  software  performance 

-  Burning  and  validating  the  SW  Installation  CD 
and  procedure 

-  Writing  the  ECR  to  release  the  SW  to 
Production 
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Configuration  Management 

•  All  work  products  are  maintained  under 
SW  Configuration  Control. 

•  All  SW  changes  are  tracked  using  SW 
Configuration  Management 

•  Configuration  Management  records  are 
maintained  by  SQA. 
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Results 

•  Escalations 

-  SW  Metrics  identified  SW  Installation  was  a 
major  problem 

-  Creation  and  Improving  the  SW  Installation 
procedure  reduced  escalations  drastically 
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Results 

•  SW  Helpdesk 

-  Metrics  showed  more  issues  were  being 
closed  than  new  reports 

-  New  Reports  continued  as  Operators  used 
features  that  they  didn’t  dare  try  before. 

•  SW  Deliveries 

-  Release  that  were  1  to  2  months  late  were 
now  delivered  on  time. 
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Conclusions 

•  Using  process  to  recover  from  ad  hoc 
development: 

-  Increased  Quality 

-  On-time  Delivery 

-  Improved  Customer  Satisfication 

-  Improved  Team  Morale 
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Contact  Information 

•  Contact  Info: 

-  NexSummit  LLC 

-  Don  Borcherding 

-  dborcherding@nexsummit.com 

-  908-684-8914 

-  www.nexsummit.com 
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Raytheon 


Organization  and  Accomplishments 


Raytheon  Missile  Systems,  Headquarters  Tucson,  AZ 


"'vV  Employees:  11,000 


VV  ’04  Sales:  $3.8  B 


☆  World  Largest  Appraised  SEI 
CMMI  Level  3  Organization 
December  2004 


☆ 


SW-CMM  Level  5  in 
November  2001 
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Key  elements  of  our  approach 


Raytheon 


GOAL: 

Obtain  CMMI  certification  at  level  3 
•Opportunity  for  future  growth 
•Win  discriminator  for  RMS  &  Raytheon 
•Improved  Program  performance 


*  Understand  the  importance  of  having  a  simplified,  integrated 
product  development  architecture 

*  Understand  the  need  to  create  a  detailed  plan,  agreed  to  by  all 
stakeholders,  before  beginning  execution 

*  Learn  one  approach  to  showing  value  to  programs  &  improving  their 
performance 


Sound  architecture,  agreed  to  deployment  plan,  value  to 

program 
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RMS  Roadmap  to  CMMI  Level  3 


Raytheon 


Implementation 


Baseline 


4Q01 


r 


1Q01 


Infrastructure 

4Q02 


2-4Q02 


Gap  Analysis 


Process 
Infrastructure 
Assessment 


4Q03 


CMMI  SE/SW/IPPD 
Level  3  SCAMPI 
for  Rating 


4Q03-1Q04 


3-4Q03 


Improved 

Process 


SCAMPI 

Readiness 

Reviews 


» 

4Q04 


CMMI 
Transition 
Plan 


Formal  CMMr 
Baseline 
Assessment 


and  Process 
Changes 


Improved 
Process 
Deployment 


Institutionalized 


1-3Q03 


•IPDS  /  CMMI  Dedicated  Team 
•Critical  Chain  Mgmt.  Approach 
• Created  new  architecture-IPDS@RMS 
•Still  IPDP  based 


The  whole  process  takes  time 
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CMMI  Organization 


Raytheon 
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CMMI  Level  3  Master  Schedule 


Raytheon 


1 .0  RMS  Reviews 


RMS  Process  Reviews 
Product  Line  Reviews 


2.0  Management 

Stake  Holder  Mtgs 

3.0  Appraisals 


4.0  Deployment 

Program,  Org  Increment 

5.0  Infrastructure 


IPDS  @  RMS 


6.0  ORG 


Enterprise  Performance  Council/ 
■  EpG  Report 

7.0  Training  l 

Class/Delta  Class  Development 
Strategic/Tactical  Learning  Plan 
Training  Effectiveness  Surveys 
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Simple  schedule  ? 
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Where  did  IPDS  @  RMS  come  from? 


Raytheon 


Corporate  IPDS 


( 2384  Task  descriptors) 


IPDS  @  RMS 


IPDP 

Core  Supporting  Processes 

Core 


Methods 


(  822  Task  descriptors) 


Our  goal  was  to  use  everything  we  could  from  IPDS,  but  simplify  it 
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IPDS  @  RMS  Architecture 


Raytheon 


| Business  Strategy 

Manning/Execution 


0 


IPDS  @ 
RMS 


Program  Planning, 
i  Management  and  Control 


System  Integration, 

Verification 
&  Validation — 


ENG 


PM 


O&S 


SU 


|\/lethocls|  jwiethods  \  \  ^/lethodj 

^/lethodsj  ^ethodj  ^ 


ethods 


Enablers  (e.g.,  checklists,  templates, 
tools,  training,  examples,  etc.)  support 
both  the  Core  and  the  Methods 


Each  CORE  process  represented  by 
process  flows,  task  descriptors,  and 
storyboards. 

Combination  of  the  six  Core 

processes  fully  CMMI  and  ISO 

Programs 
Plans 
•PMP 
•SEMP 
•CMOP 
Etc 


What  processes  will  be  used, 
modified,  or  added.  What 
products  will  be  included. 
Selection  of  appropriate 
methods. 


Simple  architecture 
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IPDS  @  RMS  Streamlining 
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Vetted  Detailed  Plan 


Raytheon 


•  Plans  placed  across  a  20  ft  wide  wall 

-  Wall-walks  addressed  hot  spots 

-  Stickies  used  too  allow  easy  details  adjustment 

•  Brought  most  Stakeholders  to  a  centralized  location 

-  Dedicated  meeting  rooms  next  to  core  team  members 

•  Multiple  events  provided  to  engage  Stakeholders 

-  Daily  Morning  Stand-Ups  with  Core  Team 

-  Weekly  Status  Meetings  with  extended  Core  Team 

-  Monthly  Frontrunner  lunches 

-  Monthly  Functional  Leadership  breakfasts 

-  Quarterly  Organizational  Leadership  reviews 

-  IPDS@RMS  Gate  Reviews 


Regularly  scheduled  meeting  allowed  for  quick  Communication 
and  agreement  on  Plan  modifications 


©  2000,  Raytheon  Company.  All  Rights  Reserved. 
Revised:  September,  2002  -  20446AGP 


1-12 


Page  12 


Value  to  Programs 


Raytheon 


•  Engaging  the  Stakeholders  increased  buy-in 

-  Frontrunner  Programs  instituted  new  processes,  becoming  more 
efficient  in  their  performance  execution 

-  Functional  Leadership  committed  to  provided  Subject-Matter- 
Experts  well-versed  on  IPDS@RMS  requirements 

*  Greater  understanding  of  the  intent  of  IPDS@RMS 

-  Tailor  processes  to  enhance  performance 

-  Document  tailoring  decisions 

•  More  selective  in  opportunities  to  pursue 

-  Recognize  and  walk  away  from  unprofitable  situations 

*  CMMI  Level  3  Certification 

-  Increased  Customer  confidence 


P  buy-  in  <=>  P  understanding  <=>  ^  success1^ 

*  bookings  =>  P  profitability 
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Results 


Raytheon 


•  Utilized  CMMI  Appraisal  Expertise  to  host  numerous  audits 

-  Findings  and  improvements  rolled  back  into  the  process 

•  Major  improvements  to  the  IPDS@RMS  process  content 

-  CMMI  Level  3  requirements  integrated 

•  Greater  awareness  of  IPDS@RMS  capabilities 

-  Increased  use  across  the  Organization 

•  Improvement  in  Subject-Matter-Experts 

-  Better  understanding  by  Process  Owners 

•  Training  &  Implementation  processes  improved 

-  Offerings  better  tailored  to  meet  Program  needs 

•  Improved  coordination  between  Process  experts 

-  Integrated  Program  Start  Up  Team 

•  2  Frontrunner  Programs  awarded  additional  contracts 


“If  you  build  it,  they  will  come”  Field  of  Dreams 


Raytheon 


©  2000,  Raytheon  Company.  All  Rights  Reserved. 
Revised:  September,  2002  -  20446AGP 
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Key  elements  of  our  approach 


Raytheon 


Attained  GOAL: 
CMMI  Level  3  Certification 
December  2004 


Largest  world  wide  facility  to  obtain  CMMI  Level  3 

Certification 


©  2000,  Raytheon  Company.  All  Rights  Reserved. 
Revised:  September,  2002  -  20446AGP 
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Raytheon 

Customer  Success  Is  Our  Mission 


Quality  Assurance 
Involvement  Compared  to 
Program  Results 

Jill  A.  Brooks 
Network  Centric  Systems 


Agenda 


Raytheon 


•  Introduction 

•  Software  Engineering  Institute  Insight 

•  Raytheon  North  Texas  Data 

-  Cost  Performance 

-  Schedule  Performance 

-  Quality 

•  Lessons  Learned 

•  Other  Considerations 

•  Next  Steps 


Quality  Assurance  Involvement 
Compared  to  Program  Results 


10/25/2005 
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Introduction  -  Raytheon 


Raytheon 


•  Raytheon  is  an  industry  leader  in  defense  and  government  electronics, 
space,  information  technology,  technical  services,  and  business  aviation 
and  special  mission  aircraft 

•  Network  Centric  Systems  (NCS)  develops  and  produces  mission 
solutions  for  networking,  command  and  control,  battlespace  awareness, 
and  air  traffic  management 

•  Space  and  Airborne  Systems  (SAS)  provides  electro-optic/infrared 
sensors,  airborne  radars,  solid  state  high  energy  lasers,  precision 
guidance  systems,  electronic  warfare  systems,  and  space-qualified 
systems  for  civil  and  military  applications 

•  Raytheon-specific  data  examined  for  this  presentation  draws  on  both 
NCS  and  SAS  programs  executed  in  North  Texas.  Data  is  from  software 
programs 
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Introduction  -  Raytheon  continued 


Raytheon 


•  For  NCS  North  Texas: 

-8  QE  engineers 

— 145  Software  Engineers 

-30  Programs  (including  maintenance  efforts) 


10/25/2005 


Page  4 


Introduction  -  Raytheon  continued 


Raytheon 


•  Self-assessment 

•  Software  Improvement 
Team  Formed 

/  \ 
IPI  CMM  -  based  Internal  Process  Improvement  Assessment 
RTIS  Raytheon/TI  Systems 
CMMI  CMM  Integrated 
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Introduction  -  The  Burning  Platform 


Raytheon 


•  Although  the  CMMI  introduces  Quality  Assurance  (QA)  at 
Level  2,  and  QA  is  further  expanded  at  higher  levels  of 
maturity,  QA  functions  still  have  to  “prove”  their  worth  as  QA 
is  often  viewed  as  an  “overhead”  function 
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Raytheon 


Software  Engineering  Institute  (SEI) 
Insight 


•  Quality  Assurance  is  introduced  at  Level  2  of  the  Capability  Maturity  Model 
Integrated  (CMMI) 

•  Quality  activities  are  in  all  process  areas 

•  As  organizations  move  up  the  maturity  ladder,  there  are  improvements  in  program 
performance 


SEI  ha^both  qualitative  and  quantitative  data  to  support  the  previousistatement 


Process  Analyst 


Continuous 

improvement 


Process 

deployment 


Quantitative 

Analysis 


Proactive  QE 


Reactive  QC 


The  SEI  has  collected  data  which  illustrates  the 
correlation  between  organizational  maturity  and 

improved  performance 
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SEI  Insight 


Raytheon 


•  Performance  results  summarized  by  the  Software 
Engineering  Institute,  March  4,  2005 


Performance 

Improvement 

Category 

Low 

Median 

High 

Number  of 
Data  Points 

Cost 

4.5% 

38% 

87% 

14 

Schedule 

20% 

50% 

90% 

14 

Productivity 

11% 

50% 

376% 

13 

Quality 

29% 

50% 

94% 

16 

Customer 

Satisfaction 

10% 

14% 

55% 

5 

Return  on 
Investment 

2  :  1 

3  :  1 

13  :  1 

8 

Reference:  http://www.sei.cmu.edu/cmmi/results.html 
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Raytheon  North  Texas  Data  - 
Cost  Performance 


Raytheon 


•  QE  Involvement  is  measured  as  a  percentage  of  the  total 
effort  on  the  program 

•  Cost  Performance  Index  (CPI)  is  measured  as  the  ratio  of  the 
Budgeted  Cost  of  Work  Performed  (BCWP)  to  the  Actual 
Cost  of  Work  Performed  (ACWP) 


BCWP 
CPU  - 

ACWP 
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Over  Budget  Under  Budget 


Raytheon  North  Texas  Data  - 
Cost  Performance 


Raytheon 


There  is  a  positive  correlation  between  QE  Involvement 
and  Program  Cost  (measured  via  CPI) 
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Raytheon  North  Texas  Data  - 
Schedule  Performance 


Raytheon 


•  QE  Involvement  is  measured  as  a  percentage  of  the  total 
effort  on  the  program 

•  Schedule  Performance  Index  (SPI)  is  measured  as  the  ratio 
of  the  Budgeted  Cost  of  Work  Scheduled  (BCWS)  to  the 
Actual  Cost  of  Work  Performed  (ACWP) 


SPI=  - 

ACWP 


BCWS 
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Behind  Schedule  Ahead  of  Schedule 


Raytheon  North  Texas  Data  - 
Schedule  Performance 


Raytheon 


♦ 


♦ 


Less  SQE  Involvement  More  QE  Involvement 

No  apparent  correlation  between  QE  Involvement  and 

Program  Schedule  (via  SPI) 
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Raytheon  North  Texas  Data  -  Quality  Raytheon 
Defect  Containment 


•  QE  Involvement  is  measured  as  a  percentage  of  the  total 
effort  on  the  program 

•  Defect  Containment  (DC)  is  measured  as  the  ratio  of  the 
number  of  defects  which  were  detected  “in  phase”  versus  the 
total  number  of  defects 


In-phase  Defects 

DC  =  - - - 

Total  Number  of  Defects 
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Lower  Containment  Higher  Containment 


Raytheon  North  Texas  Data  -  Quality 
Defect  Containment 


Raytheon 


There  is  a  positive  correlation  between  QE  involvement 

and  Defect  Containment 
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Raytheon  North  Texas  Data  -  Quality 
Defect  Density 


Raytheon 


•  QE  Involvement  is  measured  as  a  percentage  of  the  total 
effort  on  the  program 

•  Defect  Density  (DD)  is  measured  as  the  ratio  of  the  number 
of  defects  which  were  detected  post  delivery  versus  the  size 
of  the  product.  Note  the  Equivalent  Lines  of  Code  was  used 
to  adjust  for  programs  with  significant  amounts  of  legacy 
code 


Post  Delivery  Defects 

DD  =  - - - 

Equivalent  Lines  of  Code 
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Lower  DD  Higher  DD 


Raytheon  North  Texas  Data  -  Quality 
Defect  Density  (DD) 


Raytheon 


There  is  a  negative  correlation  between  QE  involvement 
and  Defect  Density,  which  is  a  good  thing! 
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Lessons  Learned 


Raytheon 


•  Data,  data,  data 

-  Multiple  data  repositories 

-  The  color  of  money 

□  Level  of  granularity:  QE  sometimes  counted  as  part  of  management, 
planning  and  control 

□  QE  may  perform  expanded  role  activities  (non-traditional  QE  activities) 
which  are  sometimes  counted  in  the  QE  “bucket” 
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Other  Considerations 


Raytheon 


•  Execution  of  QE  improved  (QE  productivity/efficiency) 

-  Don’t  currently  have  a  formal  metric  for  this 

-  Process  has  matured 

-  QE  staff  has  had  very  little  attrition 

-  Getting  more  “bang”  for  the  QE  buck? 


QE  productivity  /  efficiency  is  an  opportunity 

for  future  analysis 
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Other  Considerations 


Raytheon 


•  Customer  value  of  QE  involvement 

-  Don’t  currently  have  a  formal  metric  for  this  (have  customer  satisfaction 
scorecards,  but  it  is  not  clear  if  these  have  the  level  of  granularity 
required  to  examine  customer  perceived  value  of  QE  activities) 

-  QE  involvement  required  by  some  contracts 

-  QE  often  has  established  long-standing  relationships  with  customers 

-  Customers  request  QE  participation  in  various  activities 


Customer  Value  of  QE  Involvement  is  another 
relationship  to  examine 
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Next  Steps 


Raytheon 


•  Continuous  Improvement  continues... 

-  Data  repository  consolidation 

-  NCS  is  moving  towards  standardized  cost 
collection  system  with  increased  granularity 

-  Metrics  are  being  standardized  across 
disciplines:  Systems  Engineering,  Software 
Engineering,  and  Hardware  Engineering 


Although  there  is  evidence  that  increased  QE 
involvement  has  a  positive  impact  on  program  success, 
there  are  opportunities  for  improvement  of  the  data  and 

more  analysis  in  the  future! 
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Contact  Information 


•  Jill  Brooks 

-  972-344-3022 
—  Jill  A  Brooks@ravtheon.com 


Raytheon 
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Questions 


Raytheon 
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Raytheon 

Customer  Success  Is  Our  Mission 


Applying  CIMMI 
to  Services 


Gordon  Ward  Director  Of  Quality  &  R6s™  Raytheon  RIS 
Juan  Ceva,  Mark  Pumar,  Enterprise  Process  Group  Raytheon 


Agenda 


Raytheon 


•  Background 

•  Lessons  Learned 

•  History 

•  Epiphanies 

•  Approach 

•  Conclusion 


“...he  who  attempts  it  must  first  pass  the  point  of  this 
lance; "  and  so  saying  he  brandished  it  so  stoutly  and 
dexterously  that  he  overawed  all  who  did  not  know  him. 

Miguel  de  Cervantes 
Don  Quixote 


Page  2 


Background  -  Critical  Issues 


Raytheon 


•  Need  to  be  CM  Ml  (L3)  to  stay  in  Business. 

•  Service  Organization  (that  don't  fit  into  the  traditional  product 
development  model)  struggle  to  achieve  CM  Ml  L3  in  a  timely 
and  cost  effective  manner. 

•  How  does  an  organization  staffed  with  practitioners  from 
standard  (product  oriented)  high  maturity  organizations,  with 
large  range  in  its  technical  disciplines,  little  or  no  process 
dollars,  and  little  or  no  project  autonomy  achieve  a  CMMI 
level  3? 

•  Welcome  to  the  World  of  Technical  Services! 
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Background  -  Pasadena  Operations 


Raytheon 


•  Raytheon’s  Pasadena  Operations 

-  Part  of  Raytheon  Information  Solutions. 

-  Establish  in  1998  with  the  award  of  the  SDSIO  (Science  Data  System 
Implementation  and  Operations)  contract  by  the  California  Institute  of 
Technology  and  NASA’s  Jet  Propulsion  Laboratory  (JPL). 

-  Umbrella  services  contract  allowing  JPL  Managers  to  contract  directly  with 
Raytheon  for  technical  and  scientific  services. 

-  Consists  of  functional  departments  organized 
along  lines  of  businesses  reporting  to  a 
Program  Manager. 

-  Technical  disciplines  range  from  software  and 
systems  engineering  to  IT,  scientific  analysis, 
and  web  development. 
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Background  -  Challenges  to  CMMI 


Raytheon 


•  Service,  not  product-based  organization. 

-  No  traditional  end-to-end  lifecycle. 

-  Customer  directing  work  -  little  or  no  autonomy. 

-  Small  overhead  -  no  funding  for  process  support  &  development. 

•  Blurring  of  function  within  projects  and  across  departments. 

-  Project  activities  range  from  software  development  to  graphic  design. 

-  Departments  support  operations,  IT,  analysis,  and  development. 

•  Raytheon  and  Customer  culture  significantly  different. 

-  Research  versus  product  oriented. 

-  Low  process  maturity  (relative  to  industry). 

-  Small  teams  (1-2  FTE)  with  modest  funding. 
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Process  Improvement  History 


Raytheon 


•  Grass  root  effort 

-  Started  with  (unfunded)  special  interest  group  in  March  1999. 

-  Prevailing  feeling  that  Customer  would  be  better  served  if  proven 
process  could  be  applied  to  each  task. 

-  Most  members  worked  for  high  maturity  organizations  before  coming  to 
Pasadena. 

•  False  starts  and  dead  ends  1 998  -  2003 

-  Top-down  approach  to  process  improvement  (unsuccessful) 

-  Obstacles: 

•  CMM,  no  CMMI  yet 

•  Lack  of  process  infrastructure  (QA,  CM,  MA,  etc.) 

•  Customer  “owns”  project  areas  (PP,  PMC,  IPM,  Risk) 

•  Small,  diverse,  short-term  (  <  year)  projects 

•  Customer  chooses  not  to  perform  some  key  practices 

•  Funding 
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Process  Improvement  History  (cont.) 


Raytheon 


•  Alternatives 

-  Approach:  Restrict  process  to  one  or  two  projects  where  sufficient 
autonomy  existed  to  apply  Model. 

-  Drawbacks:  No  benefit  on  most  projects;  limited  benefit  to  customer. 

-  Approach :  Apply  Model  to  a  collective  group  of  projects,  with  no  single 
project  performing  all  key  practices  of  the  Model. 

-  Drawbacks:  Risky  -  might  loose  one  or  more  key  projects. 

-  Approach:  Completely  new  (non-traditional)  approach. 

-  Drawbacks:  Not  clear  how  to  proceed;  requires  ‘breakthroughs . 
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History  (cont.) 


Raytheon 


•  Breakthroughs  2003  and  2004 

-  Use  bottom-up  approach. 

•  Develop  lifecycle  based  on  how  work  is  actually  done. 

•  Map  Model  to  lifecycle. 

•  “Fill”  process  gaps. 


-  Shift  focus  to  delivery  of  service  instead  of  delivery  of  a  product. 

-  Use  Raytheon  Six-Sigma  for  process  improvement. 

Raylheon  Six  Sigmai' 

-  Use  a  traditional  engineering  lifecycle  (requirements,  design, 
implementation,  verification  and  validation)  to  develop  the  process. 


-  Employ  an  evolutionary  or  staged  approach  to  implementation. 
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Epiphanies 


Raytheon 


•  Key  breakthroughs  that  resulted  in  substantial  progress  in 
applying  the  CMMI  to  services. 

•  Epiphany  1 :  Task  Orders  are  equivalent  to  projects. 

-  Apply  process  to  every  task  order. 

-  Re-identify  each  Model  key  practice  with  an  equivalent  practices  in  the 
service  environment. 

•  Epiphany  2:  Project  requirements  are  services  requested  by 
the  Customer. 

-  Requirements  are  the  Customer  requests  for  services  in  the  form  of 
personnel  and  attendant  support. 
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Epiphanies  (cont.) 


Raytheon 


•  Epiphany  3:  Every  project  has  the  same  (unchanging)  five 
requirements. 

-  Requirements  are: 

•  Staffing  -  e.g.  supply  two  oceanographers. 

•  Facilities  -  e.g.  provide  office  space  and  equipment  for  assigned  personnel. 

•  Finances  -  e.g.  monitor  and  report  cost  associated  with  supplying  2 
oceanographers. 

•  Management  -  e.g.  manage  the  task  order  (find  staff  and  facilities,  monitor 
cost  and  personnel). 

•  Infrastructure  Support  -  e.g.  provide  networks,  phones,  computers,  etc.  for 
2  oceanographers. 
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Epiphanies  (cont.) 


Raytheon 


•  Epiphany  4:  The  relative  time  spent  for  development  versus 
delivery  in  services  is  reversed  from  that  of  products. 

-  Products:  most  of  the  effort  is  spent  developing  the  product. 

-  Services:  most  of  the  effort  is  spent  delivering  the  service. 


Product 

Development 

i 

Deliver 

Service 

(Development 

Deliver 

_ i 
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Approach 


Raytheon 


•  Template-based  solutions. 

-  Technical  staff  and  management  not  burdened  with 
process  details  that  are  not  directly  applicable  to 
their  work. 
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—  Technical  staff  works  with  solutions  that  are 
relevant  to  their  everyday  tasks  without  having  to 
become  versed  in  the  CMMI. 

-  A  relatively  small  group  of  Model  experts  can 
concentrate  on  insuring  that  the  CMMI  practice 
areas  are  covered  via  the  usage  of  the  templates. 
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Approach  (cont.) 


Approach  Used 

•  Multi-part  SCAMPI  C  and  B. 

—  SCAMPI  C  divided  into  two  events. 

•  1  st  event  examined  organization’s 
business  model. 

•  2nd  event  examined  templates. 

-  SCAMPI  B  divided  into  three  events. 

•  1 st  event  examined  evidence  from  the 
project  area  of  the  Model  on  a  single 
focus  project. 

•  2nd  event  examined  evidence  from  the 
support  area  of  the  Model. 

•  3rd  event  examined  evidence  from  the 
engineering  area  of  the  Model. 

-  Engaged  appraisal  team  in 

improvement  process 

•  Team  recommendations,  solutions,  and 
feedback  incorporated  into  process  before 
deployment. 

-  SCAMPI  A:  Traditional 


Raytheon 


Traditional  Approach 

•C  and  B  SCAMPIs  are  conducted 
as  single  events 

—SCAMPI  C  reviews  policies  & 
procedures 

-SCAMPI  B  reviews  polices, 
procedures  and  artifacts 
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Lessons  Learned 


Raytheon 


Use  Bottoms-up  Approach 

-  Develop  process  solutions  based  on  business  model 

-  Tailor  solutions  to  the  organization 


Run  the  implementation  as  a  (serious)  project 

-  Establish  a  project  manager,  budget,  schedule,  and  measurable  goals. 

-  Track  and  monitor  progress  on  a  regular  basis  using  EVMS. 

-  Use  phased  deployment 

-  Develop  and  validate  processes  before  deployment. 

Implement  a  ‘ grass  roots’  communications  plan  throughout  the 
project 

-  Start  communication  with  staff  and  management  early  to  establish  and  clarify 
goals. 

-  Celebrate  small  successes  publicly  at  all-hands  meetings 
and  other  group  events. 

-  Setup  recurring  open  houses  and  training  sessions 
with  the  process  developers. 

-  Demonstrate  the  benefits  to  individuals. 


.'J  Countdown  tg 

SCAMPI  A 


Lewi  3  Certification! 
1 0- NOV-2004 
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Lessons  Learned  (cont.) 


Raytheon 


*  Obtain  stakeholders’  support  and  active  involvement. 

-  Gain  sponsors  at  the  highest  level  and  understand  their  goals. 

-  Involve  Customers  frequently  via  EPG  and  Steering  Committee 
meetings. 

-  Communicate  the  benefit  of  reaching  goals. 

•  Make  use  of  consultants. 

-  Leverage  Model  expertise  from  other  parts  of  the  organization. 

-  Choose  the  lead  appraiser  wisely  i.e.  ‘out-of-box’  thinker. 

-  Put  the  appraisal  team  to  work  for  the  organization. 

-  Use  their  feedback  to  refine  processes  before  deployment. 
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Conclusions 


RayHieon 


•  Successfully  implemented  one  of  the  first 
applications  of  the  CMMI  Model  to  Services  _ 

-  Technical  Details  Documented  in  (up-coming)  -sa,  fs.'rn.  «7-  as-®!. 

SEI  Technical  Note  '  '  "  * 

-  https ://bscw.sei.cmu.edu/pub/bscw.cqi/0/79783  'm* 


•  Developed  pragmatic  and  cost  effective 
model  to  deploy  CMMI  in  non-traditional 
applications 

•  True  Process  Improvement 

-  Added  processes  meaningful  and  useful  to 
organization 

•  Day-to-day  operations  improved 

•  Organization  more  efficient  and  effective  at 
delivering  services 

-  Increased  engagement  with  customer  at  all 
levels 
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CMMI  AS  A  SAFEGAURD 
AGAINST  SOFTWARE 

ENTROPY; _ 

MANAGER'S  PERSPECTIVE 


Dr.  Thomas  F.  Christian  Jr. 
Director  of  Engineering 
ACSSW,  WPAFB  OH 
16  Nov,  2005 


SOFTWARE 

ENGINEERING 


G  Software  Engineering  is  part  of 
Systems  Engineering 

G  Systems  Engineering  is  the 
disciplined  application  of  tools 
and  principles  to  achieve  a 
complex  goal 


G  Systems  Engineering  must  obey 
the  Fundamental  Laws  of  „  9  «.,> 
Physics  ® 


THE  LAWS  OF 
PHYSICS _ 

G  But  what  are  the  fundamental 
laws  of  physics? 

□  F=MA? 


G  Earlier  seminal  work  at  SSTC 
2005  said  "Yes"  -  Newton's 
Laws  of  Motion  govern  Software 
Engineering 


THE  LAWS  OF 

METAPHYSICS _ 

□  "I  think  therefore  I  code" 

□  No,  No!! 

G  Even  EEs  understand  Inertia 
but,  could  there  be  some  law 
even  more  fundamental  than 
that?  o— ^ 


THE  LAWS  OF 
NATURAL  PHILOSOPHY 


THERMO 

The  unexplainable 


1st  LAW  -  CONSERVATION 
OF  ENERGY 


Aq 


Heat  Transfer 

ao  c  aq 

AS  =  Entropy  = — j~ 


(COLD) 


"If  the  state  of  a  system  is 
changed  by  applying  work  or 
heat  or  both,  then  the  change  in 
the  energy  of  the  system  must 
equal  the  energy  applied" 


2nd  LAW  -  TENDENCY 
TOWARD  EQUILIBRIUM 


"It  is  impossible  to  move  heat,  by 
cyclical  process  from  something 
at  lower  temperature  to 
something  at  higher  temperature 
unless  work  is  added  to  the 

system" 


ESI 


IS 


s 


3S 
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3rd  LAW  -  ABSOLUTE 
ZERO 


"If  the  entropy  of  each  element  at 
absolute  zero  can  be  taken  as  zero, 
then  all  elements  above  absolute 
zero  must  have  a  finite,  positive 
entropy;  however,  because  entropy 
cannot  be  reduced  to  zero  by  finite 
means  (as  per  the  Second  Law),  no 
system  can  reach  absolute  zero" 


Since  we  are 
Software  Engineers 

^ _ 

Let's  put  them  into  Software 

Engineering  -  Speak 

D  1st  Law:  You  Can't  Win  - 

Just  Break  Even 

G  2nd  Law:  You  Can  Only  Break 

Even  at  Absolute  Zero 

G  3rd  Law:  You  Can't  Reach 

Absolute  Zero 


THE  SOFTWARE  LAWS  OF 
THERMODYNAMICS 


G  Optimizing  software  quality, 
cost,  schedule  require  proper 
processes,  planning,  and 
people 

D  Proper  processes,  planning  and 
people  requires  time  to  do  it 
right 

G  There  is  NEVER  time  to  do  it 
right 


"RASSA'S  LAW" 

I  l"No  one  can  resist  the 

temptation  to  edit  another's 
work  or  start  coding  on  the 
first  day  of  a  program" 
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NDIA  CMMI  Technology 
Conference  &  User  Group 


Denver,  CO 


CMMI 

•  ^ 


A  Change  Agent  in  a 
Level  1  Organization: 
How  to  Survive  in  a  Hostile 
Environment 


MASPli 


Alin 
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Agenda 


■  Introduction 

■  LI  Organization  =  Hostile  Environment? 

■  Understanding  Resistance 

■  Challenges  to  Change  in  LI  Organization 

■  How  to  be  a  Change  Agent  in  a  LI  Organization 

■  Creative  Ways  of  Measuring/Advertising  Success 

■  Summary 


Alin 
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ABB 


CO 


Leader  in  power  and  automation  technologies 

Enable  utility  and  industry  customers  to  improve 
performance  while  lowering  environmental  impact 

The  ABB  Group  of  companies  operates  in  more  than  120 
countries  and  employs  approximately  120,000  people 

ABB  became  the  first  company  in  the  world  to  sell 
100,000  robots 


A  vast  majority  of  products  at  ABB  have  software  and 
hardware  comoonents 


Alin 
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ABB’s  Organizational  Structure 


■  Power  Technologies 

■  Power  Systems 

■  Medium-Voltage  Products 

■  High  Voltage  Products 

■  Transformers 

■  Utility  Automation  Systems 

■  Automation  Technologies 

■  Automation  Products 

■  Manufacturing  Automation 

■  Process  Automation 


ASPI 


ABB 


SOFTWARE 

PROCESS 

INITIATIVE 


■  ABB  Software  Process  Initiative  (ASPI) 

■  ASPI  is  composed  of  members  from  2  ABB  Corporate 
Research  Centers  (CRCs): 

■  United  States:  Raleigh 

■  Sweden:  Vasteras 

■  Responsible  for:  Development  of  appraisal  and 
improvement  methodologies,  evaluation  and  deployment 
of  pilots  within  ABB  for  CMMI  transition,  PSP/TSP,  etc. 


LO 


D) 

0 

DC 

o 

DC 


O 
c n 
=) 
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LI  Organization  =  Hostile  Environment?  - 1 


■  Why  might  a  level  1  organization  be  considered  a 
Hostile  Environment? 

■  LI  Organizations  don’t  plan  or  monitor  well 

■  Firefighting  is  the  norm  -  TENSION! 

■  No  time  for  instituting  change 
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LI  Organization  =  Hostile  Environment?  -  2 


■  Why  might  a  level  1  organization  be  considered  a 
Hostile  Environment? 

■  Heroes  are  key  to  success 

■  The  change  may  be  seen  as  a  threat 
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LI  Organization  =  Hostile  Environment?  -  3 


■  Why  might  a  level  1  organization  be  considered  a 
Hostile  Environment? 

■  Middle  managers -Top-10  or  none 

■  Changes/improvement  efforts  typically  don’t  make  the  top-10  list 


CO 
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Managing  Resistance  - 1 


Why  People 
Resist 


When  Does 
Resistance 
Increase? 


•  Maintain  Status  Quo  and  avoid  transition  state 

•  Protect  individual  and  organizational 

•  Values 

•  Emotions 

•  Ways  of  operating 


•  Low  perceived  need 

•  Implies  poor  past  performance 

•  High  level  of  disruption 

•  Low  reward  /  high  cost 

•  Negative  consequences 


•  Irreversible  outcome 

•  Doubt  about  success 

•  Fear  of  unknown 

•  Unclear  expectations 

•  and  Low  involvement 


AH 

nn» 
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Managing  Resistance  -  2 


How  People 
Resist 


•  “Can’t  do  it!”:  Skill  (Training  Issue) 

•  “Won’t  do  it!”:  Motivation  (Management  Issue) 


2  Types  of 
Resistance 


•  Overt 

•  Covert 


* 
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Forms  of  Resistance  - 1 


■  Give  me  more  detail:  Continual  requests  for  more 
information;  no  matter  how  much  you  give,  it’s  never  enough 
(appraisal  interview  experience) 

■  Flood  YOU  with  detail:  More  and  more  information  is  provided 
that  you  understand  less  and  less. 

■  Time:  They  never  have  enough  time  to  meet  with  you,  meetings 
that  you  do  have  are  continually  interrupted  by  calls  or  by  people 
who  “drop  by”. 

■  Impracticalitv:  The  person  keeps  reminding  you  that  they  live 

in  the  Real  World. 

■  I’m  not  surprised:  No  matter  what  bizarre  and  unexpected 
things  happen  in  a  project,  they  claim  they  are  not  surprised. 


Alin 
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Forms  of  Resistance  -  2 


■  Attack:  You  are  attacked  with  angry  words,  a  red  face,  pounding 
on  the  desk,  pointing  a  finger  in  your  face,  and  punctuating  the  end 
of  every  sentence. 

■  Confusion:  They  claim  to  be  continually  confused,  even  after 
you  have  explained  things  two  or  three  times. 

■  Silence:  No  reaction  or  response,  even  when  you  push  hard  for 
concurrence  or  objections. 

■  Intellectualizinq:  The  person  wants  to  discuss  theory  after 
theory  about  why  things  are  the  way  they  are. 

■  Compliance:  They  totally  agree  with  you  and  eagerly  wants  to 
know  what  to  do  next.  No  reservations  are  ever  expressed;  the 
implication  is  that  whatever  you  do  is  fine. 

■  Pressing  for  solutions:  They  want  to  rush  headlong  into 
solutions,  without  spending  the  time  necessary  to  clearly  identify 

and  analyze  the  problem(s).  H  || 

MINI 
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Reaction  Pattern  to  Change  viewed  as  Negative 


Time 
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Managing  Complex  Change 


■  Vision 

Skills 

Skills 

Vision 


Incentives 


Incentives 


Incentives 


Resources 


Resources 


Resources 


Action 

Plan 


Action 

Plan 


Action 

Plan 


Change 

Confusion 

Anxiety 


Vision 


Skills 


Resources 


Gradual 

Change 


Vision 

Skills 

Vision 

Skills 

Incentives 


Incentives 


Action 

Plan 


Resources 


Frustration 


False 

Starts 
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Challenges  to  Change  in  LI  Organization  - 1 


■  Resources 

■  Lack  of  sufficient  resources  leads  to  frustration 
in  bringing  about  change 

■  Budget  for  instituting  a  change  should  be  established  and 
supported  just  as  the  budget  for  a  development  project 


Alin 
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Challenges  to  Change  in  LI  Organization  -  2 


■  Training 

■  Not  a  high  priority  for  Level  1  organizations 

■  But  training  is  typically  a  key  component  in 
rolling  out  any  change 

■  Sponsor 

■  A  level  1  sponsor  presents  one  of  the  biggest  challenges 

■  The  sponsor  must  support  and  “own”  the  change  before  the 
organization  will  accept  the  change 


Alin 
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Challenges  to  Change  in  LI  Organization  -  3 


■  Don’t  reward  bad  behavior  -  even  the  sponsor’s! 

■  “Managers  Behaving  Badly” 

■  Rewarding/Responding  to  bad  behavior 
encourages/perpetuates  it! 

■  Sponsors  that  rule  by  intimidation,  cursing,  and  general 
instability  are  not  in  support  of  positive  change 
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How  to  be  a  Change  Agent  in  a  LI  Organization  - 1 


■  “Contract”  with  your  Sponsor 

■  Understand  the  Sponsor’s  expectations  of  the  change  effort 

■  Understand  what  the  change  effort  needs  from  the  Sponsor  in 
order  to  be  successful 

■  Create  a  “contract” 

■  Reach  a  common  understanding  with  the  Sponsor  of  his/her 
expectations  and  also  what  is  required  from  the  Sponsor. 

■  Does  not  necessarily  result  in  a  formal/legal  ‘contract’ 

■  However,  the  resultant  commitments  should  be  documented  and 
approved  (e.g.,  include  in  Process  Improvement  Plan) 
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How  to  be  a  Change  Agent  in  a  LI  Organization  -  2 


■  “Re-Contract”  with  your  Sponsor 

■  “Re-Contracting”  is  necessary  when  support  from  your  Sponsor 
is  less  than  committed,  (the  improvement  effort  is  being 
negatively  impacted) 

■  It  is  the  responsibility  of  the  Change  Agent  to  have  a  meeting 
with  the  Sponsor  to  resolve/re-negotiate. 

■  Usually  the  Sponsor  needs  only  a  reminder. 

■  Offer  to  “ghost  write”  drafts  of  communications  if  this  would  be 
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How  to  be  a  Change  Agent  in  a  LI  Organization  -  3 


■  The  lifeguard  approach 

■  Let  the  organization  move  forward  on  their  own 

■  Make  sure  they  follow  the  rules 

■  Jump  in  if  the  effort  is  floundering 

■  Demand  (at  least)  Level  2  behavior  from  those 
responsible  for  change 

■  “Practice  what  you  preach” 


Alin 
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How  to  be  a  Change  Agent  in  a  LI  Organization  -  4 


■  Ride  the  bull 

■  You  may  go  through  some  difficult  times  when 
instituting  a  change,  but  hold  on  tight, 
be  persistent. 


■  “If  ya  ride  the  bull,  you’re  going  to  get  some  bruises!” 


■  Know  when  to  step  back 

■  Important  that  the  organization,  not  just  the  change  agent  is 
passionate  about  the  change 


■  It’s  not  about  you  doing  all  of  the  work 

■  Be  a  catalyst  and  provide  support 

■  The  organization  needs  to  make  the  change  happen 


Alin 


©  ABB  USCRC  Raleigh  -  22 


How  to  be  a  Change  Agent  in  a  LI  Organization  -  5 


■  Don’t  underestimate  the  power  of  brochures  and  posters 

■  Constant  reminders  keep  awareness  of  the  change  at  a  high 
level 

■  Training  “on  the  cheap” 

■  Lunchtime  seminars 

■  Newsletters 

■  Promotions 

■  Lollipop  tree 

■  Questions  -  Donuts 

■  PowerPoint  presentation  cycling  in  the  main  lobby 
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Ways  of  Measuring/Advertising  Success  - 1 


■  The  importance  of  low-hanging  fruit 

■  Pick  the  fruit  while  it’s  ripe 

■  Share  with  others 

■  Don’t  take  credit  for  the  tree  or  the  fruit 

■  Make  a  pie  when  appropriate 

■  Set  it  on  the  window  sill  to  cool  so  that  all 
of  the  neighbors  can  enjoy 

■  Open  up  a  fruit  stand 


Alin 
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Creative  Ways  of  Measuring/Advertising  Success  -  2 


■  Reward  and  Recognition 

■  You  don’t  have  to  spend  a  fortune 

■  Little  things  mean  a  lot 

■  Posters 

■  “Turtles” 

■  “Traffic  lights” 

■  Dare  to  be  different! 


Alin 
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Wrap-up/Summary 


■  Being  a  Change  Agent  is  sometimes  just  a  matter  of 
survival  and  the  opportunity  to  seek  shelter! 

■  However,  the  toughest  jobs  are  also  the  most  rewarding. 

■  Being  a  Change  Agent  is  not  for  the  timid  or  shy. 

■  By  bringing  a  bit  of  creativity  and  a  lot  of  determination 
and  patience,  you  can  prevail! 

■  Being  a  “Target”  at  least  ensures  that  you  get  a  lot  of 
attention.  ;-) 


Alin 
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Contact  Information 


Andy  Cordes 

ABB  USCRC 

940  Main  Campus  Drive 

Raleigh,  NC  27606 

phone:  (91 9)  856-3871 
email: 

andrew.cordes@us.abb.com 


Dennis  Brantly 

Wachovia  Information  Technology 
1 525  W  WT  Harris  Blvd 
Charlotte,  NC  28262 

phone:  (704)  427-0823 
email: 

dennis.brantly@wachovia.com 


Strategic  Planning: 
Selling  a  CMMI-based 
Improvement  Effort  to 
Senior  Management 
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Aldo  Dagnino  and 
Andy  Cordes 


ABB  Inc. 

US  Corporate 
Research  Center 
Raleigh,  NC 
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Agenda 


■  ABB  Overview 

■  Selling  CMMI-based  Improvement  to  Senior 
Management 

■  Business  Unit  Level 

■  Business  Area  Level 

■  As  a  Strategic  Technology  at  the  Division  Level 

■  Supporting  CMMI-based  Improvement  as  a  Strategic 
Technology 

■  Summary 
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ABB  Overview 


CO 


Leader  in  power  and  automation  technologies 


Enable  utility  and  industry  customers  to  improve 
performance  while  lowering  environmental  impact 


Alin 


The  ABB  Group  of  companies  operates  in  more  than  120 
countries  and  employs  approximately  1 10,000  people 

ABB  became  the  first  company  in  the  world  to  sell 
100,000  robots 


A  vast  majority  of  products  at  ABB  have  software  and 
hardware  comoonents 
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ABB’s  Organizational  Structure 


Power  Products 


Automation  Products 


xi- 


Corporate  Research 


a  mi 

fMPIP 


Robotics 


Power  Systems 


Process  Automation 


a  mi 
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ABB’s  Products 


LO 


A  »» 


■  Power  Products 

■  Power  Systems 
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ABB’s  Products 


CD 


■  Automation  Products 

■  Process  Automation 

■  Robotics 
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ASPI 


ABB 


SOFTWARE 

PROCESS 

INITIATIVE 


■  ABB  Software  Process  Initiative  (ASPI) 

■  ASPI  is  composed  of  members  from  2  ABB  Corporate 
Research  Centers  (CRCs): 

■  United  States:  Raleigh 

■  Sweden:  Vasteras 

■  Responsible  for:  Development  of  appraisal  and 
improvement  methodologies,  evaluation  and  deployment 
of  pilots  within  ABB  for  CMMI  transition,  PSP/TSP,  etc. 
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Evolutionary  Approach  to  Selling  CMMI-based  Improvements  in  ABB 


00 


■  First  phase: 

■  CMMI  sold  at  the  level  of  local  product  development  units 

■  Second  phase: 

■  CMMI  sold  at  the  level  of  Business  Area  within  a  geographic 
region  as  a  project 

■  Third  phase: 

■  CMMI  sold  at  the  Division  level  as  a  strategic  technology 


Progress 


Sold  at  Local  Unit 
level 


Sold  at  BA 
level 


Sold  at  Division 
level 


Time 

- ► 
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CMMI  Sold  at  Local  Product  Development  Units 


Characteristics  of  effort: 

■  At  the  beginning  of  the  Process  Improvement  Program,  the  selling  effort 
was  focused  on  local  development  units 

■  Units  are  relatively  small  organizations 

■  No  history  of  CMMI-related  benefits  within  ABB  was  available 

■  CEPG  needed  high  level  of  training 

■  CEPG  needed  to  develop  support  tools  and  methodologies 

■  Product  development  projects  always  have  priority  over  process  improvement  activities 

■  Lessons  learned 

■  Commitment  highly  dependent  on  local  organizational  changes 

■  Commitments  are  short-term  due  to  annual  budget  constraints  and  short-term  changes  (I’m  not 
sure  what  you  mean  by  “short-term  changes”) 

■  High  degree  of  flexibility  within  the  organization  to  make  changes 

■  High  budget  constraints  of  local  development  units  (is  this  covered  in  the  second  bullet?) 

■  No  synchronization  of  improvement  activities  and  solutions  with  other  units  in  the  same  group 

■  Commitment  to  process  improvement  based  primarily  on  sponsor’s  beliefs  rather  than  business 
objectives 

■  Need  to  constantly  monitor  commitment  from  sponsor  and  organization 
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CMMI  Sold  at  Business  Area  (BA)  within  a  Geographic  Region 


Characteristics  of  effort: 

■  A  BA  consists  of  clusters  of  development  units 

■  CMMI-based  improvement  was  sold  as  a  unifying  activity  to 
the  BA  managers  within  geographic  regions 

■  Process  improvement  activities  were  viewed  a  projects  that  compete  for 
resources  with  product  development  projects 

■  CMMI-based  activities  sold  as  projects  competing  with  product  development 
projects 

■  Lessons  learned 

■  Commitment  to  CMMI  not  as  highly  dependent  on  organizational  changes 

■  Commitment  to  process  improvement  based  more  on  business  benefits 

■  Need  to  have  a  portfolio  of  documented  benefits  of  CMMI-based  process 
improvement  efforts 

■  Commitment  to  CMMI-based  improvement  medium-term 

■  Some  level  of  coordination  among  development  units  within  the  region 
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CMMI  Sold  at  the  Division  Level  as  a  Strategic  Technology 


■  Characteristics  of  effort 

■  Clear  business  objective  needs  to  be  defined 

■  Commitment  is  sold  at  high  Senior  Management  level 

■  Process  improvement  is  considered  as  a  program  not  a  project 

■  Senior  Management  supports  program  at  the  global  Division  level 

■  Longer  term  commitments  are  established 

■  Process  improvement  is  seen  as  competitive  advantage 

■  Process  improvement  is  not  as  dependent  on  changes  in 
organizational  structure 

■  Local  development  units  receive  funding  and  objectives  from  higher- 
up  in  the  organization 
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Process  to  Sell  CMMI  as  a  Strategic  Technology 


Process  for  any  Strategic 
Technology 


Application  to  CMMI-based 
Process  Improvement 


Selection  of  STP  lead  from  CEPG 

I 

Engagement  of  CEPG  members  and 
potential  partners  in  Development  Units 

I 

Identify  new  developments  in  CMMI  and 
other  process  improvement  related  fields 

1 

Identify  benefits  within  ABB  both 
quantitative  and  non-quantitative 

1 

Analyze  current  improvement  projects 
with  BUs  and  their  impact.  Assess  current 
collaborations 


Develop  a  plan  for  the  future  and  outline 
the  importance  of  CMMI  as  a  strategic 
technology 
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Use  of  the  ABB  Gate  Model  for  CMMI-based  Process  Improvement  Projects 


ABB  Gate  model  used  on  all  projects  to  ensure  that: 

■  Projects  are  linked  to  strategy  and  business  requirements 

■  Projects  are  executed  in  control  of  the  management 

■  Project  investment  is  structured  in  phases  to  minimize  risk 

■  Projects  are  visible  and  transparent  to  the  organization 

■  Project  deliverables  have  the  right  quality 

■  Projects  are  delivering  the  benefits  as  promised 


CMMI-based  improvement  projects  follow  the  Gate  Model  as  well 

■  Keeps  the  focus  on  the  business  benefits  of  the  improvement  effort 

■  Actively  involves  management 


Alin 
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Supporting  CMMI-based  Process  Improvement 
as  a  Strategic  Technology  -  Knowledge  Base 


■  One-stop  web-based  source 
for  Product  Development 
Resources  and  Best 
Practices  -  Organized  by 
CMMI  Process  Area 

■  Target  Audience:  Change 
Agents,  QA,  Project 
Managers 

■  Monthly  reminder  e-mails 
listing  new  additions 

■  Top  contributors  recognized 

■  Weekly  metrics  collected 
and  analyzed  to  gauge  the 
effectiveness  of  the 
knowledge  base 
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Supporting  CMMI-based  Process  Improvement 
as  a  Strategic  Technology  -  Newsletters 


■  Purpose: 

■  “Provide  insight  into  good 
product  development 
practices” 

■  Issued  quarterly  via  rich-text  e- 
mail 

■  Concise,  easy-to-digest 

■  Contents: 

■  Conference  reports 

■  Brief  summaries  of  new 
technologies 

■  Successful  ABB  development 
practices 

■  Development/Process 
Improvement  cartoon 

■  Etc. 


Development  Practices 
Newsletter 

July  2004  ■  Volumw  1,  Issue  2 

Contents 
1.  Welcome 

S.  9th  European  Software  Engines  ring  Process  Oruup  Conference  Report 

3.  Security  In  Produds 

4.  Agile  Software  Development  In  Largs  Organisations 

5.  Food  lor  Thought 

£'  Subscription  Information 


MASPII 


»  Welcome 

Welcome  io  the  Development  Practical  Nawilattar.  Thin  newsletter  will  be  published  on  a  guartarly  bails  and  will  provide 
Insight  Into  good  product  development  practices  for  ABB  tmployaas  ossoclotad  with  product  davalopmanl. 


▼  9th  luropumi  Saftwetru  E n i n« e rl n ej  Process  Group  Conference  Report 


EUROPEAN  SEPC  2004 


Tha  ninth  Europaon  Software  Engineering  Process  Group  conference  was  held  In  London  In  June.  Almost  400  attendees  had 
the  opportunity  to  take  part  in  a  prog  ram  that  In  cl  u  dad  a  Pro|atfManogam*nt  Symposium,  a  Metrics  Symposium,  a  number  of 
tutorials,  and  tha  main  conference  sessions. 

Three  themes  were  observed:  security  and  safety,  engineers  and  process  Improvement,  and  finally  metrics.  All  three  themes 
are  also  connected.  Security  Is  of  high  Importance  for  many  applications.  In  the  conference,  It  was  obvious  that  she  bonking 
sector  Is  In  need  of  more  secure  solutions.  Safety  puts  similar  demands  on  software  development  and  the  conclusion  that  many 
draw  from  the  problems  Is  that  products  need  la  be  designed  With  security  and  safety  In  mind  This  leads  la  the  fact  that  each 
engineer  needs  1o  understand  the  imparlance  of  quality  and  be  Involved  In  the  process  Improvement  efforts.  A  key  for  this  Is 
to  define  and  use  the  right  metrics,  useful  also  for  the  engineers  to  that  the  Individual  can  get  valuable  feedback  on  the  work 
nerfnrmeH 
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Lessons  Learned 


■  Selling  CMMI  as  a  Strategic  Technology  at 
higher  levels  in  the  organization  increases  the 
probability  of  success  of  the  effort 

■  It  is  essential  to  make  a  business  case  for 
CMMI-based  improvements  to  sell  them  to 
Senior  Management 

■  Tracking  the  economic  benefits  of  CMMI-based 
improvements  is  essential 

■  Think  Global  and  act  Local  brings  the  best  of 
both  worlds 


Alin 


Questions  ? 


©ABB  Inc.,  USCRC  - 
11/16-2005  - 


Aldo  Dagnino 


ABB  Inc. 

US  Corporate 
Research  Center 
Raleigh,  NC 


Measuring  Economic 
Benefits  of  Process 


Improvement  in  CMMI 
Level  1  Organizations 


Alin 
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ABB  Overview 


■  Leader  in  power  and  automation 
technologies 

■  Enable  utility  and  industry 
customers  to  improve  performance 
while  lowering  environmental 
impact 

■  ABB’s  products  help  operate 
Utilities,  process  industries, 
manufacturing  plants,  and  other 
industries 


■  Present  in  over  1 20  countries 
and  employs  1 10,000  people 

■  First  company  in  the  world  to 
sell  100,000  robots 

■  A  vast  majority  of  ABB  products 
have  software  &  hardware 

components 
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ABB’s  Organizational  Structure 


Power  Products 


Automation  Products 


CO 


Corporate  Research 


a  mi 

fMPIP 


Robotics 


Power  Systems 


Process  Automation 


a  mi 
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ABB’s  Products 


A  »» 


■  Power  Products 

■  Power  Systems 


ABB  Inc.,  USCRC  - 


ABB’s  Products 


LO 


■  Automation  Products 

■  Process  Automation 

■  Robotics 


Alin 


ABB  Inc.,  USCRC  - 


ASPI 


ABB 

SOFTWARE 

PROCESS 

INITIATIVE 


CD 


ABB  Software  Process  Initiative  (ASPI) 
acts  as  the  Corporate  Engineering 
Process  Group 

ASPI  is  composed  of  members  from  2 
ABB  Corporate  Research  Centers 
(CRCs): 

■  United  States:  Raleigh 


■  Sweden:  Vasteras 


Responsible  for: 

■  Initiation  activities 

■  Performance  of  appraisals 

■  Development  of  improvement 
methodologies, 

■  Evaluation  and  deployment  of  pilots  within 
ABB  for  CMMI  transition,  PSP/TSP,  etc. 

■  Assisting  units  in  establishing  improvement 
plans  and  acting 

■  Collect  lessons  learned  from  process 
improvement  activities 


Alin 
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ABB  Corporate  EPG  Support 


Support  ABB  Development  Units  in  their  Continuous  Improvement 
Efforts  to  establish  a  culture  of  product  development  excellence 


CEPG 


SECRC  ASPI  Team 
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Continuous  Process  Improvement  Cycle 


co 


Initiate  Improvement  activity 

■  Define  Medium/Long-term 
Strategic  Improvement  Plan  (SIP) 
and  identify  organization’s 
business  goals 

Conduct  internal  CMMI  Appraisal 
(Class  B) 

Develop  Process  Improvement 
Plan  (PIP) 

■  Prioritize  process  improvement 
activities  using  Business 
Objectives 

Implement  PIP  and  monitor 
Lessons  learned 


1 

Stimulus  for  Change 

A 


Learning 


Diagnosing 


Acting 


Establishing 


Re-Initiate 


Alin 
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Process  Improvement  Driven  by  Competitive  Advantage 


■  Primary  customers  of  ABB  are  commercial  organizations 
(Utilities,  petrochemical  industries,  pharmaceutical, 
automotive,  chemical  plants,  etc.) 

■  Motivation  to  improve  is  driven  by  business  reasons 

■  When  Maturity  Level  is  not  a  business  objective,  prioritization 
of  improvement  activities  is  paramount 


Increase 

Competitive 

Advantage 


A  llll 

mm 


Process  Improvement 
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Results  of  Internal  ABB  Class  B  CMMI  Appraisal 


■  Establishes  a  baseline  in 
the  organization 

■  Serves  as  a  basis  to 
identify  process 
improvement  activities 

■  Recommended  to  include 
the  Measurement  and 
Analysis  Process  Area 


Generic  Goal  2 


Practice 

RD 

ReqM 

PP 

PMC 

MA 

SAM 

Ver 

PPQA 

CM 

Specific  Goal  1 

SP1.1 

Medium 

Medium 

High 

High 

High 

High 

■ 

High! 

SP  1.2 

High 

Medium 

High 

High 

High 

High 

Highl 

SP  1.3 

High 

High 

High 

High| 

SP  1.4 

High 

SP  1.5 

High 

High 

SP  1.6 

High 

SP  1.7 

High 

Specific  Goal  2 

SP  2.1 

High 

Medium 

Medium 

High 

High 

High 

High 

SP  2.2 

High 

High 

High 

High 

High 

SP  2.3 

High 

High 

High 

High 

High 

SP  2.4 

High 

High 

High 

SP  2.5 

Medium 

SP  2.6 

High 

SP  2.7 

Specific  Goal  3 

SP  3.1 

Medium 

High 

High 

High 

SP  3.2 

Medium 

High 

■■ 

SP  3.3 

High 

High 

SP  3.4 

High 

SP  3.5 

Medium 

Alin 
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Measurement  and  Analysis 


■  Two  Types  of  metrics: 

■  Metrics  associated  with  the  product 

■  Metrics  associated  with  the  development 
process 
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Typical  Process-related  MA  in  a  CMMI  Level  1  Organization 


■  Measurement  Objectives  for  Process  Improvement  not  clearly 
defined 

■  Information  needs  and  objectives  are  not  consistently  defined  and 
documented 

■  Measurement  objectives  are  not  consistently  defined 

■  Measurement  objectives  are  not  consistently  aligned  with 
information  needs 

■  Specify  Measures  for  Process  Improvement 

■  Quantifiable  measures  are  not  consistently  traceable  to 
measurement  objectives 

■  No  clear  definition  between  base  and  derived  measures 


■  Collection  and  storage  of  specific  measurement  data  associated 
with  process  improvement  is  not  consistently  defined 


■  Analysis  and  reporting  of  measurement  data  for  process 
improvement  is  not  consistently  specified 


© 


Alin 
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Goal-Questions-Metrics  Paradigm 


■  GQM  presents  a  systematic  approach  for  integrating 
goals  to  models  of  the  software  processes,  products  and 
quality  perspectives  of  interest  based  upon  the  specific 
needs  of  the  project  and  the  organization.  (Basili  et  at, 
1994). 


Alin 
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GQM  Definitions 


■  Define  major  goals  of  the  process  improvement  activity 


■  Questions  derived  from  goals  that  must  be  answered  to 
determine  if  the  goals  are  achieved 


■  Measurements  that  provide  the  most  appropriate 
information  for  answering  the  identified  questions 
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Example  of  GQM  for  Process  Measurement 


Goals 


Questions 


Metrics 


Decrease 
Development  Costs 


How  many  defects 
products  have? 


Number  of  defects 
found  in  released 
products  in  first 
three  months,  by 
month 


Improve  Quality  of 
Products 


Ensure  on-time 
Delivery 


Decrease  Risks  in 
Projects 


Alin 
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Product  Development  Process  Metrics 

■  Typically  associated  with: 

■  Consumption  of  resources  during  a  process 

■  Process  control 

■  Errors  or  faults  associated  with  a  particular  process 
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Example  of  Development  Process  Metrics 


■  Management  control  metrics 

■  Deviation  between  actual  and  estimates 

■  Deviation  from  promised  final  delivery 

■  Test  coverage  metrics 

■  Number  of  defects  introduced 

■  Cost  of  reducing  defects 

■  Where  defects  are  introduced 

■  Error  distribution  by  cause 

■  Effort 

■  Person/time  metrics  (not  elapsed  but  actual) 

■  Time 


■  Time  to  market  metrics 


■  Productivity 

■  Software  output  per  unit  of  input 


Alin 


Discussion  of  an  Example  at  ABB 


Please  refer  to  Handouts  to  follow 
specific  Example  discussion 
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Lessons  Learned 


■  A  CMMI  appraisal  provides  the  foundation  for  process  improvement 

■  Using  the  GQM  approach  is  a  useful  way  to  establish  a  metrics 
program  for  process  improvement 

■  Establish  Goals  from  business  objectives 

■  Business  objectives  should  be  employed  to  prioritize  process 
improvement  activities  after  appraisal  has  been  conducted 

■  Use  the  CMMI  Measurement  and  Analysis  process  area  practices 
to  establish  metrics  for  process  improvement 

■  Process  improvement  should  include  the  MA  process  area  together 
with  any  other  improvement  to  ensure  meaningful  measurements 
are  obtained 

■  Start  small  and  simple 


Alin 


Questions  ? 


Customer  Success  Is  Our  Mission 

Finding  CMMI  Compliant  Artifacts  and 

A  Needle  In  A  Haystack 


Presented  to  National  Defense  Industrial  Association 
Fifth  Annual  CMMI  Technology  Conference  and  User  Group 
Denver,  Colorado 

Adrio  DeCicco 

Space  and  Airborne  Systems 
Raytheon  El  Segundo,  CA 
November  17-20,  2005 


l^TncBton 

You  Are  Tasked  To  Find  CMMI  Compliant 
Data  For  An  Upcoming  Appraisal 

•  You  meet  with  some  program  leads  who  show  you  the  program  server 
and  where  their  documents  are 

•  You  start  with  Project  Planning  (PP)  and  Project  Monitoring  and  Control 
(PMC) 

-  You  begin  to  locate  documents  for  the  each  of  the  process  areas 

-  You  create  a  matrix  and  populate  the  matrix  with  documents  for  each  of  the 
process  areas 

-  You  cannot  find  data  for  some  practices,  so  you  question  the  leads 

PP  GP  2.10,  Review  the  activities,  status,  and  results  of  the  project  planning  process 
with  higher  level  management  and  resolve  issues 

The  program  manager  is  not  available,  however  the  leads  provide  you  with 
program  reviews 


11/23/2005 
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You  Continue  To  Collect  Artifacts 

•  -  Some  examples  of  your  data  collection  activities: 

-  For  PP,  SP  2.7  Establish  and  maintain  the  overall  project  plan  content: 

You  provide  the  program  management,  systems,  software,  risk,  and  quality 
plans. 

-  For  PMC,  SP  1 .1  Monitor  the  actual  values  of  the  project  planning 
parameters  against  the  project  plan: 

You  look  at  the  examples  suggested  in  the  model’s  sub-practices  and  you 
provide  the  program  schedule  and  a  schedule  of  resources 

-  For  PMC,  SP  1 .3  Monitor  risks  against  those  identified  in  the  project 
plan: 

You  provide  a  list  of  hardware  and  software  risks 

-  For  PMC,  SP  1 .6  Periodically  review  the  projects  progress, 
performance,  and  issues: 

You  provide  a  design  review 
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You  Completed  Data  Collection  And  Are 
Ready  For  The  Appraisal 

-You  feel  good  about  what  you  have  done  and  expect  to  get 
favorable  results  from  the  upcoming  appraisal 

-Collecting  data  was  a  lot  of  work  but  you  felt  it  was  not 
difficult 
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The  Appraisal  Results  Are  Not  Favorable 

-  There  are  many  observations  from  the  appraisers 

For  PP,  SP  2.7  Establish  and  maintain  the  overall  project  plan  content: 

You  provided  the  program  management,  systems,  software,  risk,  and  quality 
plans 

»  Some  plans  are  not  signed 

»  There  is  no  evidence  of  “maintain”  as  all  plans  are  a  Rev  (-) 

For  PMC,  SP  1 .1  Monitor  the  actual  values  of  the  project  planning  parameters  against 
the  project  plan: 

You  provided  the  program  schedule  and  a  schedule  of  resources 
>>  The  schedules  are  not  updated  nor  show  actuals 
For  PMC,  SP  1 .3  Monitor  risks  against  those  identified  in  the  project  plan: 

You  provided  a  list  of  hardware  and  software  risks 

>>  The  risks  are  not  the  same  ones  shown  in  the  program’s  risk  register 
For  PMC,  SP  1 .6  Periodically  review  the  projects  progress,  performance,  and  issues: 
You  provided  a  design  review 

>>  It  is  not  applicable  here.  Move  it  to  SP  1 .7,  Review  the 
accomplishments  and  results  of  the  project  at  selected  project 
milestones.  Provide  other  program  or  team  reviews,  not  milestone 
reviews 
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More  Observations  From  The  Appraisers 

•  Plans  are  not  direct  evidence  unless  shown  in  Project 
Planning 

•  List  of  action  items  does  not  describe  the  closure  activity 

•  Reviews  with  higher  level  management  do  not  show  line 
management  attended 

•  GP  2.5  Train  the  People:  the  training  records  of  the  people 
shown  are  not  the  same  as  the  org  charts  (GP  2.4)  or  charge 
numbers  people  used  in  providing  resources  (GP  2.3) 

•  ISO  audit  does  not  show  evidence  of  objective  evaluation 
because  it  does  not  include  many  of  the  sub-practices  of  the 
subject  process  area 

•  Little  systems  engineering  evidence  was  provided  for  at  least 
one  program 

•  Data  is  “old” 

•  The  SAM  process  appears  to  have  stopped  two  years  ago 
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Fallout  From  The  Appraisal 

•  You  are  now  convinced  that  finding  CMMI  compliant  data  is 
like  finding  a  needle  in  a  haystack! 

-  As  a  matter  of  fact,  it  is  more  like  finding  a  needle  in  Field  of  Hay! 
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Sorting  Out  The  Needle  From  The  Hay 


•  Prior  to  an  appraisal  understand  what  is  expected  from  the 
appraisal  team  and  what  you  expect 

-  What  constitutes  “enough”  artifacts 

Two  directs  and  two  indirects  (per  practice,  sub-practice,  discipline,  etc.) 

-  What  constitutes  compliancy  to  the  model 

Meet  the  intent  of  the  goal 
Provide  evidence  for  all  practices 
Provide  evidence  for  all  sub-practices 

-  Educate  the  appraisal  team  on  how  your  business  /  organization  operates 

Provide  overviews  from  each  appraisal  program  and  your  organization 

Size,  life  cycle,  what  is  being  built  and  how  and  by  whom  (disciplines), 
etc. 

-  Communicate  your  goals 

Need  help  to  improve  a  certain  business  organization,  discipline,  etc. 

11/23/2005  I  Page  8 


Sorting  Out  The  Needle  From  The  Hay  (Cont’d) 

•  Define  terms  for  data  collection  guidance 

-“Goodness”  -  appraise  plans  for  CMMI  compliancy  and  do  not 
judge  the  effectiveness  of  the  plan 

-How  “old”  is  “old?” 

-Understand  “maintain”  in  “establish  and  maintain” 

-“Update” -changed  plans,  requirements,  milestones,  events 

—“Refresh”  -  recurring  activities  (meetings,  monitor  and  control, 
etc.) 
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Sorting  Out  The  Needle  From  The  Hay  (Cont’d) 

•  Know  how  the  appraisal  team  will  operate 


-Will  credit  be  given  for  artifacts  seen  in  prior  practices  but  not 
provided  in  other  practices  where  they  also  apply? 

-Will  appraisers  external  to  your  organization  ask  appraisers 
from  your  organization  to  help  clarify  data  issues? 

-Similarly,  will  appraisers  ask  for  help  from  other  appraisers  to 
help  clarify  data  issues  from  a  discipline  they  are  not  familiar 
with? 
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Sorting  Out  The  Needle  From  The  Hay  (Cont’d) 

•  Describe  how  documents  are  organized  for  the 
appraisal 

-Spreadsheet,  table,  by  discipline,  etc. 

-Soft  and  /  or  hard  copy  of  data 

-How  classified,  proprietary,  etc.  data  will  be  provided 

-Data  collection  threads 

—Comments  explaining  why  the  data  is  relevant 
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PM  Plan 
SE  Plan 
SW  Plan 


PM  Plan 
CM  Plan 
RSK  Plan 


MA  Plan 
VAL  Plan 
OT  Plan 


VER  Plan 
OT  Plan 
SW  Plan 


Using  A  Mixture  Of  Documents  Makes  It  More  Difficult  To  Understand  The  Evidence 


Documentation  Issues 


PM  Plan 
SE  Plan 
SW  Plan 


PM  Plan 
SE  Plan 
SW  Plan 


PM  Plan 
SE  Plan 
SW  Plan 


PM  Plan 
SE  Plan 
SW  Plan 


Some  Issues/Considerations  That  Resulted  From  Appraisals: 


-  PMC:  Monitor  Effort,  Cost,  And  Schedule  Against  The  Plan 

-  GP  2.8  (Monitor  &  Control):  Monitor  &  Control  The  Process  Against  The  Plan 

-  GP  2.9  (Objective  Eval):  Process  Is  Implemented  As  Planned 

-  PPQA,  SP  2.1:  Can  Be  Little  “q”;  Not  Necessarily  Big  “Q” 

-  GP  2.10  (Higher  Level  Mgt):  Should  Have  Actual  Presentations  &  Meeting  Minutes 

Attendee  List,  &  Action  Items 

Higher  Level  Management  Is  Senior  Management;  Not 
Program  Management 

-  Documentation:  How  Much  Is  Enough? 

Two  Directs  &  Two  Indirects? 

Need  Directs  &  Indirects  From  PM,  SE,  &  SW? 

Are  Plans  Indirects? 
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Other  Issues 

•  Implementing  CMMI  Based  Process  Improvements  On  Legacy 
Programs 

-  Introduce  Process  Improvements  As  Legacy  Programs  Enter  A  Major 
Milestone 

Tailor  The  Organization’s  Standard  Process  From  That  Point  Forward 

•  Have  A  Senior  Systems  Engineer  Or  A  Senior  Software  Engineer 
Be  Responsible  For  Applicable  Systems  Engineering  Activities 
On  Predominantly  Software  Programs 

•  Time  And  Effort  Trying  To  Produce  Lessons  Learned  Artifacts  For 
Every  CMMI  Process  Area 
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Conclusions 

•  Manage  appraisal  preparation  like  you  would  a  program 

-  Assign  responsibility,  obtain  commitment  (organization,  program,  and 
process),  generate  a  schedule,  resolve  issues,  monitor  progress,  and 
report  status 

•  Determine  who  should  identify  documents 

-  Process  engineers  or  program  personnel 

•  Form  a  capable  team 

-  Process  engineers  with  CMMI  training 

-  Process  engineers  who  can  verify  that  the  collected  documents  are 
CMMI  compliant 

-  Communication  between  data  verifiers  and  collectors  to  understand 
what  is  expected  to  satisfy  the  model 

Reviews  and  meetings  need  minutes,  action  items,  attendee  lists 
Data  threads;  the  plans  shown  in  PP  should  be  the  same  in  PMC 

•  Appraisal  team 
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Summary 

•  Diverse  Opinions  Emerge  On  How  To  Define,  Generate,  And 
Implement  CMMI  Compliant  Processes 

-  Process  Engineers,  Appraisers,  And  Lead  Appraisers  Have  Divergent 
Opinions  On  The  Intent  Of  The  Model  And  What  Constitutes  Compliancy 
Detail  And  Architecture  Of  The  Organization’s  Standard  Process 
Artifact  Collection:  Type  And  Number  Of  Documents 
Scope,  Meaning,  And  Application  Of  Generic  Practices 
Correlating  An  Organization’s  Daily  Activities  To  CMMI  Practices 

•  As  A  Result,  At  Times  It  Seems  As  If  We  Looking  For  A  Needle 
In  A  Haystack 
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Background 


□  From  a  Process  Improvement  (PI)  viewpoint 

■  Three  (3)  major  sites 

□Scope  changed  since  abstract  submitted 


■  Added  one  more  large  Integrated  Product  Team  (IPT) 

□One  of  four  Mission  Area  Teams  (MATs)  in 
NAVAIR 

Built  on  gains  achieved 
by  NAVAIR  Software 
Process  Improvement 
Community  of  Practice 
(SPI  CoP) 

■  All  MAT  PI  Leads  knew  each  other  from  SPI  CoP 
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MAT  Business  Objective 


“The  ever-increasing  demand  on 
software  capabilities  requires 
NAVAIR  to  exponentially  increase 
its  ability  to  consistently  deliver 
high  quality  software  at  minimal 

cost  February  2005  MAT/NSSC  CONOPS 
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MAT  Architecture 


FROM 


Mission  Area  Team  (MAT)  Leadership 
Admin  /  Business  Ops  /  Processes 
Chief  Engineers 


Blocks  / 
Proj  Leads 


Blocks  / 
Proj  Leads 


Blocks  / 
Proj  Leads 


O  O  OO  OOOO  0  00000000,  0(3 

0  o  o  o  o 


#  *  o  ©  o  o  ®  ®  * 

TASK  TEAMS 

{skilled  workforce  building  /  integrating  products) 


TO 
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MAT  Functional  Groups 


Certain  functional  groups  will  not  flow 
between  Product  Teams,  but  we  will  develop 
communication  bridges  to  flow  information 
and  best  practices 


Other  groups  will  be  available  for  planning 
across  groups,  but  will  require  varying 
amounts  of  training  to  obtain  proficiency 


Some  groups  will  flow  rather  seamlessly 
across  the  Product  Teams 


Product  Lead<J I l/>Product  LeadC-jVProduct  Lead 

Chief  Engr  <^>  Chief  Engr  <J>  Chief  Engr 

Block  Lead  <z] 

J=^BIock/Prj  LeadC=j 

j=£lock/Prj  Lead 

Functional 
Group  A 

•••••  C; 

Functional 
Group  A 

n 

Functional 
Group  A 

••••• 

1  1 

Functional  j  Functional  J  Functional 
Group  B  i  Group  B  ■  Group  B 

•••••  •••••  ••••• 

i  i 

!  1 

Functional 
Group  C 

<^>  Communication  Bridge 


1 

Functional 
Group  n 

j 
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MAT  Overarching  Requirements 


Acquire,  develop,  maintain  tactical 
systems  software  at  reduced  cost  with 
acceptable  risk 

□Continuously  improve  quality,  productivity, 
responsiveness,  and  predictability 

□  Ensure  alignment  to  fleet  requirements 


■NAV^A  I  R- 
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MAT  SSEPG 


PI  Lead(s)  from  each  of  the  IPT  SSEPGs 

Decision  making  by  consensus  so  far 

Worked  together  in  SPI  CoP  by 
consensus  before  standing  up  MAT 

□  Equal  vote  for  each  SSEPG  member 

Just  hired  SSEPG  Lead  to  interface  with 
MAT  Leadership  team 


■NAV^A  I  R- 
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Aligning  PI  with  CMMI 

□  NAVAIR  Initiative 

□NAVAIR  products  integrate  software  and 
systems  functions 

CMM  ignored  integi 


□  Using  SEI  IDEALSM 


planning,  ana 
implementing  MAT 


PI 
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Aligning  PI  with  CMMI2 


Continuous  representation  and  equivalent 
staging  OR  one  huge  SCAMPI 

■  Determine  process  areas  that  apply  to  each 
IPT  according  to  mission 

■  Ensure  MAT  coverage  across  Maturity 
Level  3 

■  Perform  Class  C  gap  analysis  for  each  IPT 

■  Fill  holes 

■  Conduct  SCAMPI(s) 
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Chronology  &  Successes 


MAT  SSEPG  allowed  to  self-organize 

■  Biggest  success  enabler 

■  Facilitation/change 
management  enabled  progress 

■  No  major  roadblocks 

■  Each  IPT  donated  PI  Lead  time  to  standup 
MAT  SSEPG 


v 


■  Three  co-located  PI  Leads  (already  meeting) 
expanded  to  include  representatives  from  all 
seven  (7)  IPTs 
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Chronology  &  Successes- 


□  Meeting  1  -Forming  MAT  SSEPG 

■  Each  PI  Lead  introduced  self  and  described  PI 
initiatives  within  their  IPT 


■  Draft  charter 

■  Consensus  decision  making  process 

■  Agreed  that  if  IPT  PI  Lead  did  not  attend  or  send  rep 
they  would  live  with  the  decisions  made 

□  Meeting  2  -  Initiating  SSEPG 

■  Each  PI  Lead  expressed  ideas  for  what  SSEPG 
should  be 


Decided  need  SSEPG  Lead  and  set  meeting  to  write 
job  description 

- NAV^A  I  R- 
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Chronology  &  Successes; 


□  Rotated  representation  to  MAT 
Leadership  meetings  (May  -  Aug) 

Used  virtual  meetings 

□  Each  IPT  rewrote  PI  Plan  for  4th  quarter 
FY05  aligned  with  MAT  objectives 

□Leader  appointed  9/30/2005 


■NAV^A  I  R- 
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Chronology  &  Successes, 


Strategic  Plan  -  PI  objectives  aligned  with 
MAT  objectives 

■  Meeting  to  produce  first  draft 

■  Comments  submitted 

■  Online  Peer  review 

■  WG  proposed  some  final  changes 

■  Baselined  08/25/2005 

Some  members  attended  special  Change 
Management  workshop 


NAV^A  I  R- 
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Chronology  &  Successes; 


□Tactical  planning 

■  Stakeholder  matrix 

■  Brainstormed  improvements  to  handle  first 

•  Each  IPT  prioritized 

•  Added  up  scores  of  all  the  IPTs  and  used  to  get 
initial  order 

•  Proposed  some  changes  in  order  which  were 
voted  and  approved  by  consensus 
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Next  Steps 


□  Finish  initial  planning 

Begin  work  on  highest  priorities 

□  integrate  PI  with  AIRSpeed? 

■  Customer  Value  Added  Definition 

■  Collect  Measures  that 

quantify  PI  savings  — i 
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Lessons  Learned 


□Getting  all  PI  Leads 
involved  early  enabled  us  to 
function  as  a  team  earlier 

□  Resistance  melted  with 
communication 

Change  Management  Facilitation  allowed 
us  to  concentrate  on  the  PI 


■NAV^A  I  R- 
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Questions? 


NAV^A  I  R 


Backup  Slides 


NAV^A  I  R 


AIRSpeed 

IDEALSM 

IPT 

MAT 

NAVAIR 

PI 

SPI  CoP 
SSEPG 

TOC 


Acronyms 

Lean  Six  Sigma  and  Theory  of  Constraints 

Initiating,  Diagnosing,  Establishing,  Acting,  and 
Learning 

Integrated  Product  Team 
Mission  Area  Team 
Naval  Air  Systems  Command 
Process  Improvement 

Software  Process  Improvement  Community  of 
Practice 

Systems  and  Software  Engineering  Process 
Group 

Theory  of  Constraints 


■nav^a  i  R- 
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Other  NAVAIR  Initiatives 


AIRSpeed  (Lean  Six  Sigma  with  Theory  of 
Constraints) 

Organizational  Change  Management 


NAV^A  I  R- 
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AIRSpeed 


Customer  (Fleet)  value-added  viewpoint 

■  Eliminate  waste  and  time  waiting  on  another 
process  (Lean) 

■  Ensure  all  steps  are  necessary  (Lean) 

■  Streamline  processes  to  eliminate  bottlenecks 
and  maximize  throughput  (TOC) 

■  Eliminate  rework  by  controlling  process  (6o) 

□  Realize  real  savings  and  recapitalize  for 
investment  in  future 


'JAV^A  I  R- 
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Organizational  Change  Management 


□  Enabler  for  both  CMMI  and  AIRSpeed 
□Train  leaders  in  change  management 
□Assess  current  state 

■  Awareness 

■  Resistance/Obstacles 

■  Knowledge  and  training  required 

□  Plan  change  and  implement  plan 

□  Effective  communication 

■  Sponsorship 

■  Feedback 

■  Allow  employees  to  “make  a  difference” 


nav^a  i  r- 


14-17  Nov  2005 


NDIA  5  th  Annual  CMMI  Conference 


22 


BAE  SYSTEMS 


Lessons  Learned  on  the  SCAMPI  Road  to 
CMMI-Software  Level  5 

NDIA  CMMI  Technology  Conference, 

14-17  November  2005  Denver  CO 


Joseph  N  Frisina 
Randall  J  Varga 


One  Road  Towards 


BAE  SYSTEMS 


NDIA  Road  to  SCAMPI. ppt  -  2 


BAE  SYSTEMS 

Communication,  Navigation  Identification  and  Reconnaissance  (CNIR) 


BAE  SYSTEMS 


-Formally  assessed  at  Capability  Maturity  Model  Integration  (CMMI)  Software 
Engineering  Maturity  Level  5  and  Systems  Engineering  /  Program 
Management  Maturity  Level  3  on  15  December  2005. 

-The  assessment  was  performed  using  the  Carnegie  Mellon  University  (CMU) 
Software  Engineering  Institute  (SEI)  CMMI  SCAMPI  A  Appraisal  method 

-Engineering  and  program  management  organizations  were  located  across 
three  states. 

-The  presentation  will  describe  the  planning  and  associated  activities  that  led  to 
this  successful  result  and  the  lessons  learned  from  those  activities  that  were  then 

cycled  into  a  continuing  process  improvement  activity. 
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Road  to  CMMI  Software  Level  5 


BAE  SYSTEMS 


-We  developed  a  database  approach  to  the  collection  and  control  of  CMMI 
artifacts  which  proved  to  be  a  valuable  resource  during  the  SCAMPI  Assessment. 

-BAE  Systems  Software  had  been  previously  assessed  at  CMM  level  5,  and  we 
developed  transition  approaches  to  the  more  comprehensive  CMMI 
representation. 
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What  is  SCAMPI  ? 


BAE  SYSTEMS 


SCAMPI  -  What  is  it? 


BAE  SYSTEMS 


SCAMPI 
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Standard  CMMI  Appraisal  Method  for  Process  Improvement 

( SCAMPI ) 


BAE  SYSTEMS 


-Ten  member  appraisal  team 

-The  appraisal  team  was  led  by  Ms.  Marilyn  Bush,  co-author  of  the  Software 
Capability  Maturity  Model 

-Team  had  4  Lead  Assessors  serving  as  members 
-Team  conducted  158  interviews 

-Team  reviewed  over  800  technical  and  management  artifacts 

-Interviews  with  all  engineering  and  program  management  organizations  located 
spread  across  three  states 
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Overview 


-Current  State 

-High  Maturity  Company 
-Goal 

-Transition  from  SW-CMM  to  CMMI 
-  Do  not  disrupt  SW-CMM  activities 

-Capitalize  on  experience  obtained  and  infrastructure 
CMMI 


Leverage  off 

- ► 

Knowledge  Gained 


BAE  SYSTEMS 


established  with  SW- 
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Basic  Decisions 


BAE  SYSTEMS 


Top-Down 


Continuous 
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What  Have  We  Learned? 


BAE  SYSTEMS 


-Capture  How  We  Do  Business 

-Processes  should  not  be  “wish  lists” 

-Get  Practitioners  Involved 

-  to  increase  the  chances  of  compliance 
-Make  processes  inclusive 

-Incorporate  Tailoring,  Links  to  Training  Materials,  Templates,  Help  Files 
-Maintain  process  on  Web  for  easy  access 

-Some  processes  already  accepted  by  other  disciplines  -  capitalize  on  that 
-Process  Team  composed  of  practitioners 
-Avoid  “Ivory  Tower”  effect 
-  Provide  rapid  response  to  update  requests 


Repeat  What  Works ! 
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Transition  Plan  From  CMM  To  CMMI  (1) 


BAE  SYSTEMS 


-  Pick  the  CMMI  Model  that  fits  your  culture 

-SW-CMM  is  staged  -  company  is  more  familiar  with  this  type  of  model 
-Use  CMMI  Staged 

-Involvement  and  Communication  Are  Key 

-Cross-Functional  Teams  of  Software,  Systems,  Programs,  CM,  Quality,  etc. 

-SEPG  Members  intimate  with  how  we  became  a  high  maturity  organization 
are  involved 

-Define  a  Core  Team  representative  of  all  stakeholders 
-  Core  Team  member  on  every  mini-team 
-Cross-Functional  Core  Team  to  Oversee 

-Cross-Functional  Mini-Teams  write  processes  and  develop  organizational  assets 
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Transition  Plan  From  CMM  To  CMMI  (2) 


BAE  SYSTEMS 


-  Look  at  What  Already  Exists 

-Some  Software  Processes  already  adopted  by  other  disciplines 

-  Expand  those  processes  to  encompass  all  appropriate  disciplines 

-  Review  Software  processes  for  potential  to  integrate  other  disciplines 
-Where  expansion  is  not  practical,  have  discipline-specific  sub-processes 

-  Maintain  existing  software  processes  as  much  as  possible 

-  Review  Other  Assets  in  the  Software  OSSP 

-They  serve  as  good  indicators  for  what  type  of  assets  will  be  required  for  CMMI 
-Templates,  training  materials,  databases,  etc. 

-Use  Existing  Software  Infrastructure  as  a  model 


Don’t  Re-Invent  the  Wheel 
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Transition  Plan  From  CMM  To  CMMI  (3) 


BAE  SYSTEMS 


-Strive  Towards  Fully  integrated  process  assets  and  infrastructure  across 
disciplines 

-Software,  Systems,  Hardware,  Programs 
-Integrated,  inter-disciplinary  Process  Development  Teams  develop  processes. 
-Templates,  Training  Materials,  Help  Files 
-Linked  directly  into  process 

-Perform  Gap  Analysis  Between  CMMI  and  our  processes 

-CMMI  Compliance  verified  via  Peer  Review  Tester  role  as  well  as 
generation/maintenance  of  a  DOORS  cross  reference  matrix 
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Transition  Plan  From  CMM  To  CMMI  (4) 


BAE  SYSTEMS 


-A  Process  Steering  Group  (PSG)  “Core  Team”  established  to  manage  changes 

-  Processes  are  integrated  -  must  assess  impact  to  other  areas  and  update 
processes  in  concert 

-Processes  reviewed  and  approved  by  Core  Team 

-  Processes/Process  Assets  on  the  Intranet  for  easy  access 

-Select  projects  for  deployment  based  on  where  they  are  in  the  Life  Cycle 

-Process  Deployment  monitored  and  tracked  against  plan  and  corrective  action 
taken  as  needed 

-Process  implementation  monitored  to  determine  process  effectiveness  and 
adjustments  incorporated 


Plan  the  Work  and  Work  the  Plan 
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Development  Organizational  Structure 


BAE  SYSTEMS 


CORE  Team 


Engineering!  Programs  ■  Operations 


Cross  functional  teams  write  the  actual  processes  and  supporting 

documentation. 
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Implementation  Organizational  Structure 


BAE  SYSTEMS 


A  “Core  Team”  maintains 
consistency  and  ensures  that 
changes  in  one  area  either 
don’t  impact  or  other 
disciplines  or  ensures  that 
impact  is  communicated  and 
addressed.  They  also  control 
the  global  (common)  site  and 
global  (common)  processes. 


Core  Team 
SW1  PM1 
SYS  1  QA1 


Individual  PSGs  deal  with 
issues  that  impact  their 
disciplines.  Representation 
allows  for  impacts  to  be 
identified.  They  control 
their  site  and  their  unique 
processes 


SEPG 

SW1 

SW2 

SW3 

SW4 

PM1 

SYS  1 

QA  1 

PMPSG 

PM1 

PM2 

PM3 

PM4 

SW1 

SYS1 

QA1 

Svs  PSG 

SYS  1 

SYS2 

SYS3 

SYS4 

SW1 

PM1 

QA1 
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Design  of  Web  Site 


BAE  SYSTEMS 


-Establish/Maintain  separate  discipline  sites 
-Establish/Maintain  integrated  site 

-Have  bi-directional  links  between  the  ‘specific”  pages  to  the  “integrated”  page  for 
an  “integrated”  OSSP 

-Support  multiple  user  view  points  I— I 

-Users  can  get  information  by 

-  Entering  the  site  for  their  discipline 

-  Entering  through  the  “main”  site  LJ 

-Modularity  allows  for  easy  growth 

-  Incorporation  of  other  disciplines  over  time 
-Add  new  discipline  Web  site 
-Add  links  from  “integrated”  site  to  new  site 

-  Do  not  need  to  go  to  every  existing  site  to  add  the  new  link,  since  all  of  the 
individual  sites  reference  each  other  through  the  “integrated”  page 
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Integrated  Web  Site  Framework 


BAE  SYSTEMS 


Best  Practices 

-  SW  BP  1 

-  SW  BP  N 

Other  BP 


Integrated 
Best  Practices 

Program  Mgt 

Software 

Systems 


Best  Practices 
-  PM  BP  1 


-  PM  BP  N 

Other  BP 


Multiple  Entry  Views  Ensure  You  Find  What  You  Want 
Regardless  of  Where  You  Start. 
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Data  Storage  Options 


Separate 

-  Each  discipline  has  its  own  database  with 
local  control 

-  Database  can  be  specifically  tailored  for 
each  discipline 

-  Difficult  to  keeping  in  concert 

-  Just  provides  needed  discipline  information 

-  Recurring  work  for  each  discipline 

-  Generating  global  status  is  difficult 


BAE  SYSTEMS 


Integrated 

-  Single  database  with  central  control 

-  Precludes  discipline-specific  tailoring 

-  Eliminates  synchronization  issues 

-  Contains  all  discipline  information  -  need  to 
be  able  to  sort  on  discipline 

-  No  recurring  cost  -  adding  new  discipline  is 
relatively  simply 

-  Can  be  set  up  to  generate  metrics  per 
discipline  and  globally  across  all  disciplines 
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Lessons  Learned 


BAE  SYSTEMS 


Having  Process  Owners 


BAE  SYSTEMS 


It’s  MINE  and 
YOU  can’t  have 


•  Practitioners  do  not  feel  they  have 
ownership 

•They  feel  it  is  imposed  ON  them 

•  Lack  of  buy-in 

•  No  real  incentive  to  provide 
feedback 


ik - * 

r 

OSSP 

< 

-JT 

SDP 

◄ — ' 

- ► 

Processes  Need  To  Be  Owned  By  The  Practitioners 
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Using  CMM  Speak 


I’m  not  sure  what  these 
things  mean  -  but  they 
sure  sound  impressive ! 


BAE  SYSTEMS 


KPAs ...  Key  Practices 
...  OSSP  ...  Managed 
Level..  Process 
Tailoring 


When  you  don’t 
know  what  you 
are  talking  about 
-  it  shows 
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Mandating  Use  of  Inappropriate  Tools 


BAE  SYSTEMS 


Imposing  inappropriate  tools 

•  Adds  no  value, 

•  Creates  more  work, 

•  Discontented  practitioners 
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Static  Processes 


BAE  SYSTEMS 


Premise  1 :  If  a  process  is  being  used,  the 
naturally  adapt  it  to  the  given  situation 


practitioner  will 


Premise  2:  If  the  infrastructure  is  in  place,  practitioners  will 
communicate  these  changes  to  the  SEPG  to  make  the 
processes  better. 


Conclusion:  If  processes  are  stagnant,  either  they  are  not 
being  used  or  there  is  no  path  to  allow  change 

If  processes  are  being  used,  the  practitioners 
will  improve  them  over  time 
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Having  Processes  That  Are  Overly  Specific 


BAE  SYSTEMS 


Processes  cannot  anticipate  all 
possible  situations 

Overly  specific  processes  cannot 
be  followed  effectively  across 
different  projects 


“One  of  the  challenges  of  Level  3  is  to  build 
processes  that  ‘empower’  the  individuals  doing  the 
work  without  being  overly  rigid.  Watts  Humphrey 
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Poorly  Defined/Confusing  Tailoring  Guidelines 


BAE  SYSTEMS 


•  Makes  it  difficult/cumbersome  to  adapt  to 
your  project 

•  Complicates  understanding 


Tailoring  guidelines  should  be  clear 
and  readily  accessible 
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Attempting  To  “Stack  the  Deck”  for  FAR  Groups 


BAE  SYSTEMS 


ATM 


Interviewee 


Doesn’t 
anyone  else  do 
anything 
around  here? 


ATM 


Overly  “Hand-selecting”  your 
best  people  as  interviewees 
prevents  some  areas  of 
improvement  from  being 
identified  -  hindering  real 
progress 

The  Assessment  Team  will 
wonder  why  the  same 
people  are  interviewed  10 
times  in  20  interviews 


Put  your  best  foot  forward 
-  but  remember  the  goal 
is  improvement 
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Not  Knowing  Where  Processes  Are  Or  How  To  Change  Them 


BAE  SYSTEMS 


How  can  you  be  FOLLOWING 
the  process  when  you  can’t 
even  FIND  THEM  ?? 
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Not  Being  Able  to  Discuss  Why  You  Do  Certain  Things 


BAE  SYSTEMS 


Why  Do  You  Do  X? 

•  “Because  the  process  says  so” 

•  “I’ve  never  thought  about  it” 

•  “Because  it’s  always  been  done 
that  way” 


If  you  don’t  know  why  you  are 
doing  something  -  FIND  OUT 

If  it  is  not  value-added,  you 
shouldn’t  be  doing  it 
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Trying  To  Sound  More  Important  Than  You  Really  Are 


BAE  SYSTEMS 


Inflating  your 
importance  can  lead  to 
credibility  issues 


Importance  Is  Like  Beauty  -  If  you  have  to  tell  people, 

you  really  aren’t. 
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Having  a  Poor/Ineffective  Site  Coordinator 


•  Distracts  Team  from  focusing  on  their  job 

•  Gives  the  first  impression  of  an  immature 
company 


BAE  SYSTEMS 
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Inconsistency  Between  Processes  and  the  Way  Work  Is  Done 


BAE  SYSTEMS 


Real 

Work 


If  they  are  inconsistent  -  then  you 
are  not  following  the  process  - 
they  are  only  for  show 
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Having  Inadequate  Resources 


BAE  SYSTEMS 


•  Appropriate  tools 
are  not  provided 


•  Training  budget 
always  overrun 


•  Not  enough  people 
allowed  to  work  on  process 
improvement 


Insufficient  budget  to 
support  needed 
activities 


You  need  the  appropriate  resources 
to  do  the  job  properly 
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Failure  To  Get  Program  Management  &  Middle  Management  Buy-In 


BAE  SYSTEMS 


S'” 


Believes  in  the  benefit  of  process 
improvement  to  the  organization 

Sponsors  process  improvement 

Believes  process  improvement  is  just 
another  gimmick 

“Don’t  bother  me  with  this  nonsense 
-  leave  me  alone  to  do  my  job” 

Believes  that  process  improvement  is 
the  right  way  to  go  and  will  help  him  do 
his  job  better 
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Associated  Groups  View  Process  Improvement  as  a  “Software” 
Thing 


BAE  SYSTEMS 


Talking  Before  Thinking 


BAE  SYSTEMS 


Engage  Brain  BEFORE  Opening  Mouth 
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Concluding  Remarks 


BAE  SYSTEMS 


-Successful  assessments  are  a  result  of  many  factors 

-  No  one  item  will  cause  you  to  pass  or  fail 

-The  overall  picture  you  present  to  the  assessment  team  will  determine  the 
outcome 

-No  Assessment  runs  perfectly  -  but  you  should  maximize  your  chance  of 
success  by  avoiding  obvious  pitfalls 

-  Say  what  you  do  and  do  what  you  say 

-  Know  why  you  do  what  you  do 

-  Be  honest  about  what  you  do 
-Remember:  goal  of  an  Assessment  is  improvement 
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BAE  SYSTEMS 


BAE  SYSTEMS 
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S' IWI lull  A _ 

The  Sh 
during. 


1)  Two  teams 

2)  Each  team  gets  a  question  posed 
to  them.  They  get  30  sec&pds  to 
answer.  If  they  a^wjpr  Xpe&fiCt  ly, 
they  get  15  points.  If  they  miss,  the 
question  goes  to^he  second  team 
and  they  get  30  se&mdsw  answer 
correctly.  /  y '  p  V  \^V\\  > 

3)  There  are  two  refunds  and  a  final 
question  where  ea^h  team  can 
answer  questions  from  the  SCAMPI  B 
and  C  Handbook. 


Question  #  1 


•  How  many  SCAMPI  vl. 
appraisals  were  conducted  from 
its  April  2002  release  through 
June  2005and  reported  to  the 
SEI? 

•  1520 

•  868 
•  545 


Question  #  1 


•  How  many  SCAMPI  vl. 
appraisals  were  conducted  from 
its  April  2002  release  through 
June  2005  and  reported  to  the 
SEI? 

•  1520 

.  868  ***** 

•  545 


Question  #2 


•  What  is  the  largest  amount  of 
Maturity  Level  reported? 


•  Level  1 

•  Level  2 

•  Level  3 

•  Level  4 

•  Level  5 


Question  #2 


•  What  is  the  largest  amount  of 
Maturity  Level  reported? 

•  Level  1  (3.5%) 

•  Level  2  (34.1%) 

•  Level  3  (29.7%) 

•  Level  4  (4%) 

•  Level  5  (19.2%) 


Question  #  3 


•  What  is  the  largest  category  o 
reporting  organizations  for  a 


SCAMPI  Class  A? 


•  Commercial/In-house 

•  Contractor  for  Military/Government 

•  Military/Government  Agency 


Question  #  3 


•  What  is  the  largest  category  o 
reporting  organizations  for  a 


SCAMPI  Class  A? 


•  Commercial/In-house  (63.4%) 

•  Contractor  for  Military/Government 
(31.7%) 

•  Military/Government  Agency  (4.9%) 


Question  #  4 

•  What  is  the  largest  organization 
type  that  reported  having  done. 
SCAMPI  Class  As? 

•  Manufacturing 

•  Defense 

•  Services 


•  Finance,  Insurance  and  Real 


Question  #  4 


•  What  is  the  largest  organization 
type  that  reported  having  done. 
SCAMPI  Class  As? 

•  Manufacturing  (25.8%) 

•  Defense  (12%) 

•  Services  (52.1%) 

•  Finance,  Insurance  and  Real 
Estate  (6.7%) 


Question  #  5 

•  What  is  the  largest  number  by 
size  of  the  organization 
reported? 

•  1-100 
•  101-200 
•  201-2000+ 


Question  #  5 

•  What  is  the  largest  number  by 
size  of  the  organization 
reported? 

•  1-100  (40.5%) 

•  101-200  (18.7%) 

•  201-2000+  (40.9%) 


Question  #  6 


•  What  new  country  just  reported 
for  the  first  time  having  had  a  . 
SCAMPI  A? 

•  Turkey 

•  South  Africa 

•  Belarus 

•  Netherlands 


Question  #  6 


•  What  new  country  just  reported 
for  the  first  time  having  had  a  . 
SCAMPI  A? 

•  Turkey  **** 

•  South  Africa 

•  Belarus 

•  Netherlands 


Round  2:  Questions  oit 
SCAMPI  B/C  Handboo 


The  Handbook  is  an  essential 
tool  for  performing  SCAMPI  B 
and  C  Appraisals. 


Question  #  1 


•  What  is  the  Optional  ‘Fidelity’ 
Scale  for  a  SCAMPI  C  ? 


Question  #  1 


•  What  is  the  Optional  ‘Fidelity’ 
Scale  for  a  SCAMPI  C  ? 

•  Low,  Medium,  High  and  Out  of 
Scope 


Question  #  2 


•  What  are  the  3  primary  phases 
of  a  SCAMPI  Appraisal? 


Question  #  2 


•  What  are  the  3  primary  phases 
of  a  SCAMPI  Appraisal? 

•  Plan  and  Prepare 

•  Conduct 

•  Report  Results 


Question  #  3 

•  True  or  False,  SCAMPI  C  does 
not  require  the  use  of  a  team. 


Question  #  3 

•  True  or  False,  SCAMPI  C  does 
not  require  the  use  of  a  team. 

•  True,  unless  you  are  Judah,  and 
consider  yourself  a  team  of  one. 


Question  #  4 


•  True  or  False,  The  minimum 
acceptable  team  size  for 
SCAMPI  B  is  4  people. 


Question  #  4 

•  True  or  False,  The  minimum 
acceptable  team  size  for 
SCAMPI  B  is  4  people. 

•  False,  the  minimum  team  size  for 
SCAMPI  B  is  2  people 


Question  #  5 


•  True  or  False:  For  SCAMPI  B 
and  C,  every  interview  can  be 
conducted  by  use  of 
teleconference  (if  interviews 
are  used) 


Question  # 

•  True  or  False: 

every  interview  can  be  conducted 
by  use  of  teleconference  (if 
interviews  are  used) 

•  True  “  For  SCAMPI  B  and  C,  there  is  no 
limitation  on  the  use  of  technology  to 
conduct  interviews.  Every  interview 
can  be  conducted  through  use  of 
teleconference  (or  video 
teleconference)  technology.” 


Question  #  6 

•  True  or  False,  For  a  SCAMPI  C, 
affirmations  if  used,  do  not  have 
to  be  corroborated. 


Question  #  6 


True  or  False,  For  a  SCA 
affirmations  if  used,  do  not 
have  to  be  corroborated. 


•  True,  the  use  of  a  single  data 
source  is  all  that  is  needed  for 
the  SCAMPI  C  method.” 


Question  #  7 


•  True  or  False:  All  team 
members  of  SCAMPI  B  and 
SCAMPI  C  must  complete  the 
Intro  to  CMMI  (SEI  Official 
course)  . 


Question  #  7 


SCAMPI  B  and  SCAMPI  C  must 
complete  the  Intro  to  CMMI  (SEI  * 
Officialcourse)  . 

•  True.  “To  serve  as  a  team  member  on  a 
SCAMPI,  one  must  successfully 
complete  the  Introduction  to  CMMI. 
This  course  must  be  taught  by  an  SEI- 
Authorized  instructor,  working  on 
behalf  of  an  SEI  Partner,  in  order  to  be 
accepted.” 


Question  #  8 


•  What  is  the  Required  Practice 
Characterization  Scale  for  a 
SCAMPI  B? 

•  Bonus  question:  What  other 
two  ‘different’  characterizations 
are  allowed? 


Question  #  8 

•  What  is  the  Required  Practice 
Characterization  Scale  for  a 
SCAMPI  B? 

•  Practice  Characterization  for 
SCAMPI  B  are  “Red,  Yellow, 
Green” 

•  Bonus:  Not  yet  and  Out  of  Scope 


Question  #  9 

•  True  of  False:  In  a  SCAMPI  B  or 
C  there  shall  be  no 
characterization  assigned  to 
process  areas,  or  to  goals 
within  a  process  area.  The  only 
model  element  that  can  have 
characterizations  is  a  specific 
or  generic  practice. 


Question  #  9 

•  True  of  False:  In  a  SCAMPI  B  or  C 
there  shall  be  no  characterization 
assigned  to  process  areas,  or  to 
goals  within  a  process  area.  The 
only  model  element  that  can  have 
characterizations  is  a  specific  or 
generic  practice. 

•  True:  Section  2.5  of  the  Handbook 
describes  this  requirement. 


Question  #10 

•  True  or  False:  An  oral 


presentation  is  sufficient  for 
SCAMPI  B  and  C  appraisal 
results  as  a  lasting  record  of 
the  outputs. 


Question  #10 


True  or  False:  An  oral  presen 
is  sufficient  for  SCAMPI  B  and  C 
appraisal  results  as  a  lasting  record 
of  the  outputs. 


•  False:  Every  SCAMPI  B  and  C  appraisal 
must  have  documented  results  that 
represent  a  lasting  record  of  the 
outputs.  An  oral  presentation  alone  is 
not  sufficient.  “If  it’s  not  documented,  it 
doesn’t  exist”. 


Final  Double  Jeopae 


•  Each  team  can  bid  all  or  none  01 
their  score. 


The  Category  is:  Generate 
Appraisal  Record. 


•  Name  4  of  the  7  (SCAMPI  B/C) 
items  that  the  appraisal  record, 
must  contain. 


You  have  2  minutes 


And  the  Answer 

•  dates  of  the  appraisal 

•  appraisal  input 

•  identification  of  the  appraisal  method  used 
(including  tailoring) 

•  findings,  including  strength  and/or 
weakness  statements 

•  practice  characterizations  (if  generated) 

•  other  characterizations  of  data  or 
attributes  of  practices  or  projects, 
generated  during  the  appraisal  (if  any) 

•  appraisal  disclosure  statement 

•  Page  72,  Section  3.4  of  the  SCAMPI  B  and  C 
Handbook. 


And  the  Winner  is..2 


•  The  owners  of  Borders  Gift 
Certificates! 


Lead  Appraisers 
Gone  Wild! 

Or  How  NOT  to  Lead  an 

Appraisal! 


Background  on  this  topic,  or  why  I 
want  all  my  peers  mad  at  me. 

i  I  have  performed  SCAMPI  As  since  2000.  Have 
performed  over  1 5. 

;  Have  performed  SCAMPI  Bs  about  10 

i  Have  performed  SCAMPI  Cs  about  20 

i  Also  am  a  candidate  LA  Observer. 

i  I  see  a  lot  of  bad  habits  and  misunderstandings 
of  the  method  during  appraisals. 

i  Some  of  the  following  are  examples  of  MDD 
violations,  some  are  just  my  pet  peeves. 


Does  anyone  out  there  read  the 

MDD? 

The  Method  Description  Document  which 
describes  the  Standard  CMMI  Appraisal  Method 
for  Process  Improvement  is  designed  to  provide 
benchmark  quality  ratings  relative  to  the  CMMI. 

The  MDD  describes  the  requirements,  activities, 
and  practices  associated  with  each  of  the 
processes  that  compose  the  SCAMPI  method. 
Precise  listing  of  required  practices,  parameters 
and  variation  limits  as  well  as  optional  practices 
and  guidance  for  enacting  the  method,  are 
covered. 


Read  the  MDD 


i  Sometimes  as  a  LA  you 
have  to  read! 

i  We  know  that  if  you  don’t 
know  where  you  are,  a 
map  won’t  help,  BUT, 

i  you  can  get  the  MDD  off 
the  SEI’s  website  at 
. .  ./pub/documents/01  .repo 
rts/pdf/1  hbOOl  .pdf 

i  LAs,  this  is  YOUR  BIBLE!! 


Understand  the  Appraisal  Plan 

MDD  Section  1 .2 

i  As  in:  ‘Yes,  you  have  to  have  one’  (1.2.6) 

J  And  ‘Yes,  the  sponsor  has  to  approve/sign 
it’  (1.2.6) 

i  And  ‘Yes,  you  have  to  identify  your 
needed  resources.  (1 .2.2)  This  is  nice  to 
do  BEFORE  the  on-site  period. 

■  When  I  ask  to  see  the  schedule,  don’t  say 
“We  are  just  going  to  see  who’s  here  this 
week  to  interview”  (logistics  1 .2.4) 


Identify  Needed  Resources 

MDD  1.2.2 

j  When  I  ask  for  proof  of  team  training  and 
training  of  the  team,  don’t  say  “We  don’t 
need  no  stinking  Intro  to  CMMI  class” 
(1.3). 

i  When  I  ask  if  the  team  as  individuals  and 
as  a  whole  meet  the  minimum  criteria, 
don’t  give  me  the  blank  stare  then  ask  me 
how  many  lbs  of  Godiva  chocolate  I  want 
(1.3.2) 


Plan  and  Manage  Logistics 

MDD  1.2.4 


i  Scheduling! 

-  Please  DO  NOT  kill  your 
appraisal  team  and  make 
the  work  days  longer  than 
9  hours.  They  will  not  like 
you.  It  also  leads  to  bad 
habits! 

-  Please  DO  NOT  have  1 6 
interviews,  all  back  to  back 
in  4  days.  Leave  at  least 

1 .5  hours  between 
interviews  for  mini-team 
consolidation. 


Plan  and  Manage  Logistics 

MDD  1.2.4 


i  If  you  want  the  team 
to  be  together  ALL 
day,  have  at  least 
water  available  for 
them.  If  you  want 
them  to  work  through 
meals,  have  meals 
brought  in.  Ask  if  any 
members  have 
specific  dietary 
needs. 


Identify  Team  Leader 
MDD  1.3.1 


i  Select  an  authorized 
SCAMPI  Lead 
Appraiser  to  serve  as 
the  appraisal  team 
leader. 

[  Verify  the 
qualifications  of  the 
appraisal  team  leader 
(experience, 
knowledge,  and 
skills). 


Identify  Team  Leader 
MDD  1.3.1 


i  The  requirements  for 
a  Team  Leader  are 
outlined  in  the  SEI 
Lead  Appraiser 
program. 

i  There  can  be  only 
one  official  appraisal 
team  leader  on  any 
given  appraisal. 


Obtain  and  Analyze  Initial  Objective 

Evidence  MDD  1 .4 

■  Verification  Versus  Discovery 


Gather  high-leverage  objective 
evidence.  The  amount  of  initial 
objective  evidence  provided  by 
the  organization  will  determine 
the  proportion  of  evidence  that 
must  be  discovered  (versus 
verified)  during  the  appraisal. 
Maximizing  time  spent  in 
verification,  versus  discovery, 
is  a  key  performance  objective 
for  the  appraisal  process. 


Obtain  and  Analyze  Objective 
Evidence  MDD  1 .4 

The  effort  required  to  conduct  a  SCAMPI 
appraisal  is  a  direct  function  of  the  amount  of 
data  available  to  the  team  at  the  beginning  of 
the  process.  Before  the  appraisal  outputs  can 
be  created,  the  team  will  need  to  verify 
objective  evidence  for  each  instantiation  of 
each  practice  within  the  scope  of  the 
appraisal. 

Leave  at  least  2-3  days  for  Analyzing 
Objective  Evidence.  Do  not  do  this  AFTER 
interviews  start. 


Prepare  Participants  1.4.1 

i  Members  of  the  organization  who  participate  in 
the  appraisal  MUST  be  informed  of  their  role, 
and  the  expectations  the  sponsor  and  appraisal 
team  have. 

i  This  is  typically  accomplished  through  a  briefing 
where  the  appraisal  team  leader  provides  an 
overview  of  the  appraisal  process,  purpose,  and 
objectives 

i  Many  time  I  hear  ‘I  can’t  make  the  Opening 
Meeting  mandatory,  the  organization  won’t  come 
anyway’,  or  ‘So  and  so  is  on  TDY,  or  they  have  a 
nosebleed  today,  or  whatever’. 


Prepare  Participants  1.4.1 


r  Not  making  this 
opening  meeting 
mandatory  makes 
you,  or  someone  on 
the  team,  explain  over 
and  over  again  at  the 
beginning  of  each 
interview  to  the 
interviewees,  why 
they  are  there. 


Perform  Readiness  Review 

MDD  1.5 

i  Determine  whether  the  objective  evidence  for 
each  process  instance  is  adequate  to  proceed 
with  the  appraisal  as  planned. 

i  Review  the  feasibility  of  the  appraisal  plan  in 
light  of  the  inventory  of  objective  evidence 
available. 

i  At  least  one  readiness  review  MUST  be 
conducted  prior  to  assembling  the  team  on  site 
for  data  collection. 

i  Again,  DO  NOT  do  this  after  INTERVIEWS  start. 


Perform  Readiness  Review 

MDD  1.5 


i  At  least  one 
readiness  review 
MUST  be  conducted 
prior  to  assembling 
the  team  on  site  for 
data  collection 

i  Again,  DO  NOT  do 
this  after 

INTERVIEWS  start. 

i  Don’t  make  me  call 
this  guy  -> 


u**OWMSTUi 


Examine  Objective  Evidence 
from  Interviews  MDD  2.1 .4 


i  This  is  not  your  old  CBA- 
IPI  Methodology.  Stop 
needless  interviewing! 
Don’t  let  this  happen  to 
you  -> 

■  Obtain  face-to-face 
affirmations  for  (1)  at  least 
one  instantiation  for  each 
model  practice  in  the 
scope  of  the  appraisal,  or 
(2)  at  least  50%  of  the 
practices  corresponding  to 
each  specific  and  generic 
goal  for  each  instantiation. 


Examine  Objective  Evidence 
from  Interviews  MDD  2.1 .4 


i  And  speaking  of 
interviews.  Do  not  let 
your  appraisal  team 
members  fall  asleep 
during  the  interviews. 

i  Make  sure  that  you 
introduce  all  members 
of  the  appraisal  team 
to  the  interviewees. 


Examine  Objective  Evidence 
from  Interviews  MDD  2.1.4 

f  Also,  do  not  ask  the  same  questions  that  have 
already  been  asked.  Pay  attention,  during  all 
questions.  “These  notes  must  cover  all  areas 
investigated  during  the  interview,  and  are  not 
limited  to  the  PAs  assigned  to  the  individual 
team  member  (i.e.,  everybody  takes  notes  on 
everything).”  (2.3.1) 

i  Do  NOT  ask  a  question,  and  then  when  the 
interviewee  is  answering,  not  write  down  what 
they  are  saying....  argh! 


Take/Review/Tag  Notes 


MDD 

■  And  speaking  of  NOT 
taking  notes....  “Every 
team  member  present 
must  take  notes  during 
interviews  and 
presentations.” 

i  Just  because  during  an 
interview,  the  questions 
being  asked  do  not 
pertain  to  YOUR  PA, 
does  not  mean  you  can 
decide  to  go  home  for  the 
day. 


2.3.1 


Take/Review/Tag  Notes 

MDD  2.3.1 

j  All  team  members  actively  take  notes 
during  all  data-gathering  sessions.  “The 
purpose  is  to  record,  verbatim,  what  the 
information  source  reveals  about  the 
implementation  of  practices  in  the  project 
or  organization.” 

i  In  other  words,  do  not  rely  on  memory 
when  you  are  tagging  your  notes.  Take 
good  notes  during  the  interviews! 


Remember, 


i  Make  friends  with 
those  you  are 
interviewing  as  well 
as  those  on  your 
team. 


Some  Closing  Notes 


i  Above  all,  be  the 
LA  everyone  wants 
to  have  come 
back! 


Last  But  Not  Least! 


'Have  fun! 


CMMI  vl  .1  for  a  Service-Oriented 

Organization 


By  Steve  Hall,  Jeff  Ricketts,  Diane 

Simpson 


16  November  2005 
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Overview 


Raytheon 


This  presentation  will  describe  how  CMMI  vl.1  was  applied  to 
Raytheon  Company’s  Information  Technology  and  Scientific 
Services  (ITSS),  a  service-oriented  company,  located  in 
Pasadena,  California  so  that  the  SCAMPI  could  be  used  to 
assess  the  process  maturity  level  of  the  organization. 


•  Key  components  to  the  success  of  any  endeavor 

•  Desire  -  to  have  a  longing  for  along  with  a  strong  intention  or  aim 

•  Drive  -  a  strong  systematic  group  effort 

•  Determination  -  the  power  or  habit  of  deciding  definitively  and  firmly 

These  were  demonstrated  by  the  ITSS  organization 
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Overview 


Raytheon 


•  Prelude  to  the  Appraisal 

•  The  Appraisals 

-  The  Plan  for  the  Appraisal  Approach 

-  Goal  of  the  Appraisal  Team 

•  The  Process  of  Team  Building 

-  Appraisal  Team  Composition 

-  Synergy 

•  Understanding  the  Business  &  Applying  the  Model  and 
Appraisal  Method 

-  The  interpretations  that  were  established  and  the  balance  that  is 
required  for  keeping  the  “spirit”  of  the  model, 

-  Maintaining  the  integrity  of  the  appraisal, 

•  Summary 
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Raytheon  Pasadena  -  SDSIO  Contract 


*  Program  information 

-  Work-Order-Based  Service  Contract 

•  Focus  is  on  Delivering  Defined  Services  and  Products 

•  End-user  of  products  and  services  is  the  customer 

•  Raytheon  staff  frequently  injected  into  projects  lead  by  customer 
working  along  side  of  other  contractors  and  customers 

-  Organized  into  6  departments  (web  services,  GPS 
applications,  IT,  Science  Systems,  Ocean  Data  Center, 
and  Remote  Sensing) 

-  Number  of  Staff:  144 

-  Award:  September  1998 

-  Type:  CPAF/CPFF  -  Contract  Work  Orders  (CWO) 

-  Period  of  Performance: 

•  Completed  5  year  Base:  (Sept/03) 

•  Executing  3+2  option  years 

-  Scope:  100+  work  orders  -  63  active 
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History  of  Process  Improvement  at  ITSS 


•  Grass-roots  effort  from  the  start  -1999 

-  Special  Interest  Group  (SIG)  formed  for  Process  Improvement 

•  The  SIG  has  helped  the  customer  with  its  own  process  improvement  initiative 

•  R6s  Introduced  as  a  process  improvement  framework  at  Pasadena  -  2000 

•  Evolutionary  Process  starting  with  CMM  -  2001  through  2002 

-  First  attempts  were  top-down 

-  Not  funded 

-  Intellectual  breakthrough 

•  Introduction  of  CMMI 

•  Bottom-up  as  the  most  effective  approach 

•  CMMI  Training  and  formation  of  a  mentoring  relationship  with  a  high 
maturity  Raytheon  site  -  2003 

•  CMMI  steering  committee  formed  with  customer  and  outside  Raytheon 
support  -  2003  -  SCAMPI  planning  begins 

•  Customer  sets  goal  to  achieve  CMMI  L2  rating  by  2005  and  L3  by  2007  - 
2003 

-  Navigation  and  Mission  Design  Section  achieved  CMMI  L2  (September  24,  2004) 
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The  Appraisals 


Raytheon 


•  The  Goal 

-  Appraise  the  maturity  of  Raytheon  Company’s  Information 
Technology  and  Scientific  Services  (ITSS)  in  Pasadena,  California. 

-  Ensure  the  integrity  of  the  appraisal  process  and  outcome 

•  The  Plan  for  the  Appraisal 

-  Conduct  a  series  of  appraisals  where  process  areas  reviewed 
align  with  the  organization’s  process  deployment  schedule. 

•  Each  appraisal  event  ranged  in  duration  from  3  to  9  days 

-  Use  the  SCAMPI  A  to  determine  the  process  maturity  level  of  the 
organization 


SCAMPI  C-1 


SCAMPI  C-2 


SCAMPI  B-1 


SCAMPI  B-2 


SCAMPI  B-3 


SCAMPI  A 


1 


1 


1 


t 


t 


i 


Jun  2004 


Jul  2004 


Aug  2004 


Sep  2004 


Nov  2004 


Dec  2004 
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The  Appraisals 
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•  The  Approach 

-  The  intent  of  the  SCAMPI  C  events  was  to  understand  the  business 
model  employed  by  the  organization 

•  These  events  were  primarily  used  as  information  gathering  through 
interview  sessions  by  the  team  with  the  practitioners 

•  In  the  effort  to  understand  the  business  model,  the  appraisal  team 
sought  to  find  answers  to  such  questions  as: 

-  What  is  the  product  and  when  does  sell-off  occur? 

-  What  is  the  management  structure  that  is  in  place? 

-  Who  plans,  assigns  and  monitors  the  work? 

-  What  are  the  technical  requirements? 

-  What  is  verified;  what  is  validated  and  when? 

-  What  are  the  work  products  and  how  are  they  controlled? 

•  In  between  information  gathering  sessions  within  each  event,  the  team 
explored  possible  parallels  between  the  business  model  and  the  CMMI 

•  As  parallels  between  the  business  model  and  the  CMMI  were  identified 
and  agreed  upon,  they  were  recorded  as  the  “Group  Memory” 

•  Feedback  was  also  provided  to  the  organization,  primarily  in  the  form 
of  recommendations  and  requests  for  information 

-  In  between  SCAMPI  C  events,  process  and  templates  evolved  to 
incorporate  feedback  from  the  appraisal  team 
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The  Appraisals 
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•  The  Approach  cont. 

-  The  intent  of  the  SCAMPI  B  events  was  to  provide  feedback  to  the 
organization  based  on  their  business  model  and  its  mapping  to  the 
CMMI 

•  Through  the  conclusions  drawn  by  the  appraisal  team  and  other 
activities  undertaken  by  the  organization  prior  to  the  first  SCAMPI  B 
event,  the  organization  concluded  their  business  model  was  primarily 
that  of  a  service  organization 

•  The  team  appraised  the  evidence  provided  according  to  the 
understandings  established  for  the  service  business  model  to  the  work 
products  created  to  deliver  the  service 

•  The  record  of  the  team’s  agreements  on  model  interpretations  was 
updated  to  reflect  additional  understandings  which  were  identified 
during  each  event 

•  Rules  of  coverage  for  contributions  to  the  organizational  repository 
were  established 

•  Weaknesses  were  identified  for  the  organization  to  address. 

-  In  between  SCAMPI  B  events,  process  and  templates  continued  to 
evolve  to  incorporate  feedback  from  the  appraisal  team 
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The  Appraisals 

•  The  Approach  cont. 

-  The  intent  of  the  SCAMPI  A  Event  was  to  appraise  the 
process  maturity  level  of  the  organization  and  provide 
feedback  on  strengths  and  weaknesses 

•  Direct  evidence  in  the  form  of  work  products  created  or 
employed  during  the  course  of  delivering  the  service  was 
appraised  in  accordance  with  the  established  interpretations  of 
the  model 

-  Established  rules  of  coverage  for  contributions  to  the 
Organizational  repository  were  applied 

•  Interviews  were  used  to  satisfy  the  SCAMPI  requirement  for 
indirect  evidence;  Interview  coverage  included  100%  of  the 
projects  and  organization 

•  Strengths  and  weakness  were  identified 

•  A  rating  was  determined 
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The  Process  of  Team  Building 

•  Appraisal  Team  Composition 

-  Team  composition  underwent  a  number  of  changes,  including 

•  Planned  “tag  team  members” 

•  Dropouts  and  replacements 

-  Changes  to  the  team  make-up  occurred  in  2  slots  of  the  9-member  team 

-  Overall,  the  impact  of  the  changes  to  team  membership  was  not  significant 
and  from  one  perspective  had  an  added  benefit 

•  The  impact  was  lessened  by  the  fact  that  the  changes  occurred 
between  events 

•  The  added  benefit  was  that  each  change  presented  an  opportunity  for 
the  team  to  validate  its  interpretations  of  the  model  as  applied  to  the 
service  business  model 

-  The  final  team  composition  included  internal  (ITSS)  employees,  external 
(Raytheon  Fullerton  employees)  and  independent  contractors 

•  3  internal  (ITSS  Pasadena) 

•  3  external  (Raytheon  Fullerton) 

•  2  external  (non-Raytheon  independent  contractors) 

•  Lead  appraiser 


10 


Raytheon 


ThalesRaytheonSystems 


The  Process  of  Team  Building 


•  Synergy 

-  5  of  the  external  members  had  previous  experience  as 
appraisal  team  members 

-  1  internal  member  and  2  external  members  came  from 
organizations  with  high  maturity  levels 

-  Having  an  appraisal  team  with  this  kind  of  experience 
resulted  in: 

•  Feedback  included  examples  of  how  organizational 
weaknesses  could  be  addressed  rather  than  just  listing  the 
weaknesses 

-  Many  of  the  examples  provided  were  implemented  by  the 
organization 

•  The  teams  ability  to  “connect  the  dots”  in  real  time  to  identify 
the  available  evidence  for  satisfying  some  of  the  model’s 
requirements 

•  Feedback  identifying  available  evidence  that  would  satisfy 
multiple  PAs  under  review 
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Understanding  the  Business  and  Applying  the  Model 


•  The  most  important  thing  we  used  in 
getting  a  handle  on  applying  CMMI  to  a 
service  based  organization  was  to 
understand  the  life-cycle  of  a  typical 
service  contract 


Once  the  life  cycle  was  defined, 
everything  fell  into  place... 
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Life  Cycle  of  a  Services  “Type”  Contract  Work  Order  (CWO) 
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Support  Process  Areas 

•  PPQA  is  performed  at  the  Org  level  while 
PMC,  M&A,  and  RskM  are  performed  at  the 
project  level,  as  expected.  These 
processes  are  maintained  throughout  the 
projects  life  cycle  like  any  other  type  of 
contract  with  status  rolled  up  and  analyzed 
at  the  Org  level. 
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Planning  Process  Areas 

•  Project  Planning  (PP)  and  Integrated 
Project  Management  (IPM)  for  a  service 
contract  is  performed  after  the  contract 
award  and  maintained  throughout  the 
projects  life  cycle  with  status  rolled  up  and 
reviewed  at  the  Org  level 
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Org  Process  Areas 


Raytheon 


*  As  expected,  Organizational  Training  (OT), 
Organizational  Process  Focus  (OPF),  and 
Organizational  Process  Definition  (OPD)  are 
performed  at  the  Org  level  (EPG  and  HR) 

*  In  addition  Supplier  Agreement  Management 
(SAM)  and  Configuration  Management  (CM)  are 
also  performed  at  the  Org  level  for  all  contracts 
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Engineering  Process  Areas 


•  As  we  anticipated,  this  is  the  area  that 
created  the  greatest  amount  of  “mind 
bending”  in  applying  CMMI  to  a  service 
organization 


Lets  look  at  the  interpretation  of  these 
PAs... 
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Requirements  Development  (RD) 


•  What  are  requirements  to  a  service  organization 
other  than  “Give  me  3  people  for  6  months”?? 

•  What  was  realized  is  the  requirements  should 
focus  on  the  services  they  provide: 

-  RTSC  Pasadena  Operations  shall  provide  effective  management 
for  the  project. 

-  RTSC  Pasadena  Operations  shall  provide  adequate  staffing  for 
the  project. 

-  RTSC  Pasadena  Operations  shall  provide  sufficient  facilities  and 
equipment  for  the  project. 

-  RTSC  Pasadena  Operations  shall  manage  project  expenses  as 
specified  in  the  project  Work  Control  Plan. 

-  RTSC  Pasadena  Operations  shall  provide  adequate 
organizational  support  for  the  project. 
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Requirements  Management  (RM) 


•  So..lf  you  have  a  set  of  requirements  that  generally  don’t  change 
how  do  you  manage  them?? 

•  Well,  what  you  manage  are  the  derived  requirements  unique  to 
the  different  projects,  recorded  in  the  project’s  Work  Control  Plan 
(WCP). 

-  Derived  from  the  customer’s  (JPL)  SOW 

-  WCP  common  template  facilitates  this  derivation 

>  This  facilitates  consistency  with  requirements  and  project  plans’ 
&  work  products 

•  The  requirements  in  the  Work  Control  Plan  are  mapped  to  the 
Organization’s  requirements  providing  the  bi-directional 
traceability 

•  Requirements  are  managed  using  a  change  control  system 
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Technical  Solution  (TS) 

•  How  would  you  apply  a  Technical  Solution  to  the  need  for 
providing  people  to  perform  a  task?? 

•  The  WCP  is  also  used  to  identify  customer  needs  to  provide  the 
service 

•  facilities 

•  finances 

•  management 

•  support 

-  DAR  is  used  identify  appropriate  personnel 

>  Do  they  need  a  web  designer  or  a  rocket  scientist? 

>  How  long  will  they  be  needed? 

•  DAR  is  also  used  to  select  an  alternate 

>  What  if  your  primary  candidate  gets  hit  by  lightning? 
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_ Product  Integration  (PI) _ 

*  The  principal  product  supplied  to  our  customer  is: 

-  Staff 

-  Supported  by 

>  Management,  IT,  Facilities,  tools,  training,  and  process 
infrastructure 


*  Product  Integration  is  the  assembly  of  the  service  solution 

-  cost 

-  WCP  (designated  facilities,  support/tools,  management  and 
staff) 


•  Management  team  and  customer  review  the  integrated 
solution  and  provides  feedback  to  the  CWO  manager. 
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Verification  (VER) 
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•  The  verification  environment  -  Described  in  task  diary 

•  Verification  procedures  -  The  Project  Development  Plan  (PDP) 
describes  the  “exit  criteria”  associated  with  each  task 

•  Peer  reviews  -  The  Work  Control  Plan  (WCP)  and  PDP  are  peer 
reviewed  using  accepted  methods  and  collection  of  statistics 

•  Verification  of  selected  work  products 

-  The  Raytheon  supervisor  managing  the  Contract  Work 
Order  (CWO)  goes  through  the  task  diary  and  verifies  each 
task  is  IMPLEMENTED  according  to  established  exit  criteria 

-  When  this  verification  is  complete,  the  customer  is  notified 
signaling  the  beginning  of  the  validation  phase 
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Validation  (VAL) 
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Validation  is  the  customer  formally  acknowledging 
that  a  task  has  been  implemented  to  their 
satisfaction.  Project  lead  marks  task  as  COMPLETE 
in  the  diary  when  this  customer  acknowledgement 
has  been  received  and  documents  when  and  how 
this  acknowledgement  was  obtained 
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•  CMMI  VI. 1  can  be  applied  to  a  Service  Organization 

-  The  interpretations  that  were  established  and  the  balance 
that  was  required  to  keep  the  “spirit”  of  the  model 

-  The  integrity  of  the  appraisal  was  maintained 

-  The  benefit  of  knowledge  sharing  from  a  more  mature 
organization  with  it’s  “sister  organization” 

-  The  development  of  a  set  of  enablers  by  Raytheon  ITSS  that 
can  be  applied  to  other  Raytheon  service  organizations  and 
increasing  the  ROI  for  this  exercise 

-  Benefits  to  the  organization  as  its  processes  continue  to 
improve 
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processMax  Presentation  Overview 

I  Ttff  PROCESS  SOLUflflN  " 


•  About  pragma  Systems  and  processMax 

•  CMMI  Measurement  requirements  & 
challenges 

•  ISO  15939  Measurement  Information  Model 

•  Implementation  Approach 

•  Summary 
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processMax  pragma  Systems 

1  THE  PROCiSS  S&WTlOti  ~  I  «T 


•  Founded  1990 

•  One  of  the  first  organizations  licensed 
by  SEI  to  perform  assessments 

•  Software  Process  Improvement 
consulting  and  assessments  for  seven 
years 

•  First  release  of  processMax  in  1998 
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Government 


Government 

Contractor 


Commercial 


U.S.  Navy 

Northrop  Grumman 

ADP 

U.S.  Army 

General  Dynamics 

Bosch 

Department  of  Labor 

Mantech 

United  Healthcare 

Veterans  Benefits 
Administration 

Teledyne  Brown 

GTECH 

National  Security 
Agency 

Dynamics  Research 

Intuit 

National  Institutes  of 
Health 

L3  Communications 

Chicago  Mercantile 
Exchange 

More  than  50  successful  independent  assessments  or  appraisals 
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processMax  What  is  processMax? 

I  THf  PJT0CE55  SAtUTlGW  “  I 


Defined  Processes 

. . .  includes  all  policies,  procedures,  guidelines,  criteria,  templates, 
and  forms  in  role-based,  step-by-step  instructions,  ready  for  use. 

. . .  fully  compliant  with  the  Capability  Maturity  Model8  for  Software 
(SW-CMM)  or  Capability  Maturity  Model  Integration  (CMMI). 

Project  Repository 

. . .  total  document  management  with  version  control,  change 
control,  and  process  history. 

Integrated  Workflow 

. . .  automatic  e-mail  notification  of  tasking  and  actions 
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processMax 

I  THE  PROCESS  SOLiiTrOW  ® 


processMax 
Organization-Project  Structure 


Organizational  Portal 

Organizational  Programs  Roles 

Templates 

Guidelines 

Criteria,  Methods,  Forms 
Plans,  Reports,  Data,  Memos 
Organizational  Training  Materials 
Organizational  Training  Records 


Organization’s  Standard 
Software  Project  Process 

♦  Project  Management  Roles 

♦  Project  Technical  Roles 
♦Templates 

♦  Guidelines 

♦  Criteria 

♦  Methods 

♦  Forms 


— ~~~q  Measurement  requirements 

P^mm  Of  SW-CMM  vs.  CMMI 


SW-CMM: 

•  Measurement  is  decentralized  and  focused  on  satisfying 
individual  KPAs 

CMMI: 

•  Focus  on  measurement  -  “no  longer  in  the  fine  print” 

•  Early  emphasis  at  Maturity  level  2  -  M&A  Process  Area 

•  Other  process  areas  (OPD,  OPP,  OPM,  CAR,  OID)  have 
significant  measurement  content 

•  Also  Generic  Practices  2.8,  3.2,  4.1 , 4.2,  5.1 , 5.2 
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CMMI:  ponderous  and  complex  for  appraisers,  engineers, 
and  managers 


There  are  approximately  200  detailed  measure-related  requirements  for 
Maturity  Level  3  and  another  60  for  Maturity  Levels  4  &  5. 
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I  THE  PHD  CiE5S  SOLUTION  ® 


Measurement  &  Analysis 

(a  Level  2  Process  Area) 


Align  Measurement  Activities 


Measurement  P  an 


Procedures 


•  Collecting  data  is  very  onerous  and  error  prone 

•  CMMI  more  explicitly  requires  measurements, 
especially  of  process-oriented  activities 

•  How  to  support  large  number  of  measures 
implied  by  the  CMMI  in  a  consistent  way 

•  Diversity  of  target  organizations  -  project  types 
and  size 

•  Need  for  customizability 

•  Measurement  collection  must  be  integrated  with 
day-to-day  work 
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Measures 
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~?77 

Typical  Situation 


processMax,  Why  ISO  IEC  1 5939  Standard? 

1  TH£  PHD  J 


•  Robust  ISO  model  meets  CMMI  requirements 
and  ensures  consistency  across  Process  Areas. 

•  ISO  Measurement  Information  Model  (MIM) 
provides  “structure  linking  information  needs  to 
the  relevant  entities  and  attributes  of  concern” 

•  Base  Measures  are  mapped  to  processMax 
data  entities 

•  Derived  Measures  are  calculated  from  Base 
Measures 

•  Indicators  are  created  from  Base  or  Derived 
Measures  to  support  Information  Needs. 
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processMax 

I  THE-  PROCESS  SOtUTTOtf  * 


Key  Relationships  in  MIM 


Information  needs 


Information 

Product 


Interpretation 


Indicator 


(analysis) 

Model 


Derived 

Derived 

Measure 

Measure 

The  outcome  of  the 
measurement  process  that 
satisfies  the  information  needs 


Explanation  relating  the  quantitative  information  in 
the  indicator  to  the  information  needs  in  the 
language  of  the  measurement  users 


Variable  assigned  a  value  by  applying  the  analysis 
model  to  base  and/or  derived  measures 


Algorithm  for  combining  measures 
and  decision  criteria 


Variable  assigned  a  value  by 
applying  the  measurement  function 
to  two  or  more  base  measures 


Measureable 

Measurement 

Algorithm  for  combining  two  or 

Concept 

Function 

more  base  measures 

Base  Measure 

Base  Measure 

Variable  assigned  a  value  by 
applying  the  method  to  one  attribute 

Measurement 

T 

Measurement 

Operations  mapping  an 

Method 

Method 

attribute  to  a  scale 

Entity 


— -1 - 

Attribute 


-  -h  - 

Attribute 


Property  relevant  to 
information  needs 


©  ISO/I  EC  2002 
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processMax 

I  THE-  PROCESS  SOtUTTOtf  ® 


CMMI  Level  3  for  Software  and  Systems 

Engineering 


Measurement  Objectives  - 18 

Approximately  equivalent  to  the  Purpose  section  of  each  Process  Area. 
They  answer  ‘Why  are  we  measuring  these  particulars  items?’  A 
Measurement  Objective  is  associated  with  one  or  more  Information  Needs. 

Information  Needs  -  23 

Correspond  approximately  to  PMC  SP  1 .1  and  GP  2.8  of  each  Process 
Area.  An  Information  Need  is  associated  with  one  or  more  Indicators, 
Derived  Measures,  and/or  Base  Measures. 

•  Indicators  -  75 

Trend  or  snapshots  relying  on  Derived  Measures  and/or  Base  Measures 

Derived  Measures  -  35 

Algorithms,  programmed  inside  a  report,  relying  on  one  or  more  Base 
Measures  or  other  Derived  Measures. 

Base  Measures  -  1 02 

Defines  what  is  to  be  measured,  its  source,  when  it  is  to  be  measured,  how 
(i.e.,  count  or  record)  it  is  to  be  measured,  and  data  constraints,  such  as 
default  values. 
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processMax  MIM  Example 

I  THf  PEOClSS  StikUTlGW  “ 


•  Information  Need  #4  Monitor  project  planning  parameters 

Supported  by  ten  indicators 

-  Example  Indicator:  Trend  of  planned  versus  actual  schedule  for 
tracked  tasks  and  milestones 

•  Derived  Measure  44  Total  date  variance  to  track  slippage  per  Milestone 

•  Derived  Measure  97  Variance  of  original  planned  date  versus  actual  date 
per  Milestone 

•  Base  Measure  64  Planned  milestone  date  and  name  per  Milestone 

•  Base  Measure  65  Actual  milestone  completion  date  per  Milestone 

•  Base  Measure  76  Planned  Start  Date,  Task  Name,  Task  Type,  and  Task 
Category  per  Tracked  Task 

•  Base  Measure  77  Planned  end  date  per  Tracked  Task 

•  Base  Measure  78  Actual  end  date  per  Tracked  Task 

•  Base  Measure  79  Actual  Start  Date,  Task  Name,  Task  Type,  and  Task 
Category  per  Tracked  Task 
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Elements  of  solution 


Senior  SEPG  Project  SQA 

Mgmt  Staff  Mgmt 


Data  is  captured  close  to  source  and  in  a  timely  way. 

-  As  user  follows  process  steps,  measurement  data  is  automatically 
gathered  by  system 

...  or  via  meaningful  query  ...  Rather  than  ask  “Is  your  risk 
exposure  high?”,  ask  “What  would  be  the  cost  if  this  risk  were  to 
occur?”  and  relate  this  to  management  reserve. 

•  As  required  or  on  a  scheduled  basis,  a  reporting  user  or  the 
system  selects  from  template  reports  and  generates  report(s)  to 
support  Information  Need(s). 

•  Report  template  accesses  Measurement  Repository  and 
retrieves  relevant  base  measure  data  and  stores  output  in 
repository. 

•  Any  other  user  can  browse  stored  report(s)  and  drill-down  to 
satisfy  Information  Need(s) 
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Example 


■'Jl  pror.t^sMdK  6,<J  -  feift.ProjBC^SAlD^fift-  Microsoft  Ihlfiincl:  EKploror  pruYldtid  by  t>uH 


Edit  View  Favorites  Tools  Help 


^  Back  *  J  [£|  ^  ,  ^5sarch  [^Favorites  ’Jfledla  $  |  \  $  ,  — 


Address  |  -_£|  http .// 10 , 1 0 ,  ]  O ,  L 57/org wsb_satd_65/pmaxenglno^cnpts /pma>. ,  pPweb- 1 ,  p ,  1  1 
|  -  *  j  |  |C|  Search  *  |  ^3  18603  blocked  |  ^  Check  +  ^  Aucof 


My  Work  f  GkistLfiiy  j  Utilities  Foil  Jar  Data 


BMiriisui  orritJhi  Reports 


Campliaiica  |  y  He 


i _ 3  Additional  Work  Products 

All  Work  Pro  dude 
LJ  Configuration  Management 
Corrective  Adlons 
LJ  Criteria 

LJ  Independent  Standards  Compliance 
.J  Measurement 

LJ  Measurement  Reports 
CJ  Meeting  Memoranda 
LJ  Memoranda 
U  Personal  Workspace 

i _ I  Planning  and  Tracking 

LJ  Process  Improvement  Program  Process. 
LJ  Process  Improvement  Program  Process. 
LJ  Process  Improvement  Program  Process. 
LJ  Process  Improvement  Program  Process. 
U  Process  Improvement  Program  Process. 
LJ  Process  Improvement  Program  Process. 
LJ  Presses  Improvement  Program  Proeeea 
LJ  Process  Improvement  Program  Process 
U  Process  Improvement  Program  Process. 
LJ  Process  Improvement  Program  Process. 
LJ  Process  Improvement  Program  Process. 
LJ  Process  improvemsnt  program  Procsss. 
LJ  Process  improvemsnt  program  Procsss. 
i_j  Process  Improvemsnt  Program  Procsss. 

i _ 3  Requlrsments 

U  Risks 

LJ  Standards  Compliance 
LJ  Sup pllsr  Management 
LJ  Support  Data 
Verification 

i _ 3  White  Papers  and  Reeearcn 


.Configuration  Management 
.Corrects  Actions 
.Instantiation  end  Upgrade 
.ISO  Measurementlnfbrmallon 
.Measurement  Reports 
.Organisational  Pohclee 
.Organisational  Process  Deflmtl 
.Organisational  Process  Focus 
.Planning  endTraddng 
.Piocess  Asset  Library 
.Projects 
.skills  Reports 
.support  Data 
.Training 


I  J  Control  Pago 

Jfistartjl  1 1 ^ )w WGSMMaw  G.O  -  T ust:»T  fSjlrbox  -  Microsoft  Outlook  j 


Control  Page 


230  |  Actions  jJ 


|Actions~~j 


2£G  |  Actions  j] 


£15  |  Actions  y 


{££4 1  Actions  jj 


?y-\  |  Actions  jJ  r 
?,?%  |  Actions  | 


?n  |  Actions  jJ 

J 

1  1,0  |  Actions  j 


211 1  Actions  j| 

£JT  |  Actions  jJ  | 

i — t™ ' — ■“ i  ^ 
'/\?  |  Actions  *| 

211  [  Actions  jj 


Z 111  |  Actions  j| 

|  Actions  jJ 
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i  Report  ^~1 
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#4  |ioo%J  "crystal* 


Trend  of  Planned  versus  Actual  Number  of 
Reviews  per  Review  Type 


Project  Name:  InterGalactic  Computer  Corporation  (1  .p.1 .1) 
Display  Date:  10/14/2005 
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Trend  of  Planned  versus  Actual  Number  of  Reviews 


In  Bn  J 


i=i 


1/2005 


2/2005  3/2005 


4/2005  5/2005  6/2005  7/2005  8/2005 


■  □ 

Planned  Actual 


9/2005 


Review  Type 


Milestone  Review 
Progress  Review 


Ld 


e 


Internet 


A 


processMax  Summary  &  Benefits 


•  CMMI  measurement  is  organized  and  centralized. 

•  Standard  set  of  measures  instantiated  for  each  project, 
with  customization  and  tailoring 

•  Data  gathering  is  automated  and  interactive  reporting  is 
provided 

•  Consistent  set  of  measures  is  used  across  the 
organization  -  process  integrated  with  tools 

•  Transformation  of  measurement  process  from  an 
onerous  task  typically  performed  by  a  small  specialist 
team  to  an  integrated  approach  where  data  is  captured 
at  source  and  often  without  any  user  effort 

•  Management  insight  through  ‘best  of  breed’  graphical 
reporting 
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Thank  You 

Thank  you  for  your  attention. 

Questions? 
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NORTHROP  GRUMMAN 

E  f  I  N  I  H  G  THE  FUTURE 

Interpreting  the 
CMMI:  It 
Depends! 

CMMI  Technology  Conference  &  User  Group 

1 4-1 7  November  2005 


Rick  Hefner,  Ph.D. 

Northrop  Grumman  Corporation 

Sree  Yellayi 

Siemens  Corporate  Research 


Background 


■  Many  organizations  struggle  with  finding  practical 
implementations  of  the  CMMI  model 

■  How  does  practice _ apply  to  my  time  and  materials 

contract? 

■  How  can  a  small  project  perform  practice _ ? 

■  Is  this  evidence  enough  for  practice _ ? 

■  The  expert  answer  is  frequently  -  it  depends! 


■  This  presentation  will  show  how  to  interpret  the  model  in  a 
variety  of  contexts,  including  small  projects,  maintenance 
efforts,  and  time  and  materials  contracts 
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Interpretation  -  The  Dictionary  Meaning 

■  To  explain  or  tell  the  meaning  of 

■  To  present  in  understandable 
terms 

■  To  conceive  in  the  light  of 

■  individual  belief 

■  judgment 

■  circumstance 
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Why  Do  Interpretation  Issues  Arise? 


■  The  CMMI  model  is  a  collection  of  industry  best-practices 

■  These  best-practices  are  based  on  an  assumed  project  and 
organizational  context 

■  These  practices  must  be  adapted  for  other  situations 

Small  projects 
Short  projects 
Maintenance  projects 

Research  &  development  (internal)  projects 
Time  and  materials  contracts 


■  To  better  understand/interpret  a  practice: 

■  Review  Process  Area  introductory  material  and  Goals  to 
understand  the  purpose  of  the  process 

■  Seek  guidance  from  someone  who  has  implemented  that 
practice  in  your  context 

■  Understand  the  fundamental  principles  behind  the  practice 
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Do  You  Have  an  Open  Mind? 


Some  practitioners  want  to  believe  the  model 
does  not  apply  to  their  situation 

■  If  it  doesn’t  apply  to  me,  I  don’t  have  to  do  it! 

Adopting  the  model  means  learning  new  ways 
of  performing 

■  Must  be  willing  to  embrace  new  ideas, 
conceive  that  other’s  approaches  may  be 
better  than  yours 


NORTHROP  GRUMMAN 


SIEMENS 


Copyright  2005  Northrop  Grumman  Corporation 


Underlying  Principles  of  CMMI 


1.  Process  discipline  leads  to  predictable  project  performance 

■  Say  what  you  do;  do  what  you  say 

■  Document  the  plans/processes 

■  Communicate  them  to  the  performers  and  stakeholders 

■  Audit  to  ensure  we  are  following  them 

2.  Conscious  choices  lead  to  better  processes 

■  E.g.,  identify  relevant  stakeholders  and  their  involvement; 
identify  work  products  to  be  controlled  and  the  control  method; 
define  validation  procedures  and  criteria, ... 

3.  Organizational  learning  improves  project  performance 

■  Capture  what  works,  and  what  doesn’t 

■  Make  rules  (policies)  to  guide  projects 

■  Define  expected  processes,  and  let  projects  tailor  them  to  fit 
Capture  work  products  and  measures,  and  learn  from  them 
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Small  Projects 


■  All  the  CMMI  practices  typically  apply, 
but  must  be  performed  in  a  highly  efficient  manner 

■  Focus  on  discipline,  not  bureaucracy 

■  With  smaller  projects 

■  Communication/coordination  is  simpler 

■  It  is  more  tempting  (but  more  dangerous)  to  abandon  discipline 

■  The  ability  to  divert  staff  to  recover  from  mistakes  is  often  less 

■  Examples  of  interpretations 

■  Plans/processes  may  be  less  detailed,  less  formal 

■  “Configuration  Control  Board”  may  simply  be  the  project 
manager 

■  Peer  review  may  be  a  “buddy  check”  by  a  single  individual 
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Short  Projects 


■  “A  ‘project’  is  a  managed  set  of  interrelated 
resources  that  delivers  one  or  more  products 
to  a  customer  or  end  user....  A  project  can  be 
composed  of  projects.  ’’ 

■  Proper  application  of  CMMI  involves  proper  definition  of 
“project”  to  fit  the  work 

■  Modern  contracts  create  tasks  of  various  sizes  and  scopes 

■  Some  are  too  short/small  to  fit  the  CMMI  definition  of  “project” 

■  These  tasks  can  be  grouped  together  to  better  fit  the  CMMI 
context  of  “project” 

■  Process  discipline  benefits  longer  projects  by  reducing  the 
risk  that  something  will  go  wrong  over  time 

■  Shorter  projects  have  to  focus  on  doing  things  right  the  first 
time,  since  little  time  is  available  for  recovery 
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Maintenance  Projects 


■  The  term  “development”  in  CMMI  does  not 
exclude  maintenance 

■  The  Engineering  process  areas  often  need 
be  interpreted  in  a  smaller  scope 

■  Example 

■  A  problem  in  the  field  requires  a  “bug  fix” 

■  The  engineer  explores  whether  the  product  is  broken  or  has 
unanticipated  new  requirements  (Requirements  Development, 
Requirements  Management) 

■  Potential  changes  to  the  design  are  considered  (Technical 
Solution) 

■  The  fix  is  incorporated  (Product  Integration),  regression  tested 
(Verification)  and  deployed  to  the  field 
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Research  and  Development  Projects 

■  Some  organizations  exclude  R&D/internal 
projects  from  their  CMMI  initiative 

■  If  you  believe  that  CMMI  is  the  fastest,  cheapest  way  to 
develop  a  product,  why  wouldn’t  you  use  it  everywhere? 

■  Guidance  about  small/short  projects  applies 
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Time  and  Materials  Contracts 


■  CMMI  applies  to  any  kind  of  work,  but.... 

■  Adopting  the  CMMI  assumes  the  project  has  the  autonomy  to 
perform  the  work  in  the  best  possible  way 

■  l.e.,  can  define  their  own  process 

■  Sometimes  the  customer  sets  limits  on  cost  and  schedule 

■  Projects  can  still  meet  the  CMMI  (e.g.,  Project  Planning),  but 
must  adjust  the  work  to  fit  the  cost  and  schedule  available 

■  Process  discipline  means  you  do  not  agree  to  a  scope  of  work 
you  cannot  hope  to  perform 

■  Sometimes  the  customer  defines  the  process  to  be  used 

■  These  processes  may  or  may  not  comply  with  the  CMMI 
(i.e.,  include  the  industry  best  practices  required  to  perform 
efficiently  and  effectively) 

■  Can  advise  the  customer  on  the  success  of  your  proven 
processes  and  the  value  of  CMMI  practices 
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Summary 


■  Many  organizations  struggle  with  finding  practical 
implementations  of  the  CMMI  model 

■  You  can  determine  how  to  interpret  the  CMMI  by: 

■  Keeping  an  open  mind 

■  Reviewing  Process  Area  introductory  material  and  Goals  to 
understand  the  purpose  of  the  process 

■  Seeking  guidance  from  someone  who  has  implemented  that 
practice  in  your  context 

■  Understanding  the  fundamental  principles  behind  the  practice 
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Contact  Information 


Sree  Yellayi 

Siemens  Corporate  Research 
755  College  Road  East 
Princeton  NJ  08540 
609-734-6583 

sree.yellayi@siemens.com 
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Rick  Hefner,  Ph.D. 

Northrop  Grumman  Corporation 
One  Space  Park  -  R2/2144 
Redondo  Beach  CA  90278 
310-812-7290 
rick.hefner@ngc.com 
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Defining  the  future 


Achieving  the 
Promised  Benefits 
of  CMMI 

Ym 

CMMI  Technology  Conference  &  User  Group 

14-17  November  2005 

% 


Rick  Hefner,  Ph.D. 

Director,  Process  Initiatives 
Northrop  Grumman  Corporation 


Background 


■  Many  organizations  have  implemented  the  Capability  Maturity 
Model  Integrated  (CMMI) 

■  Although  they  have  achieved  their  desired  maturity  level  and 
improvement  goals,  some  organizations  have  seen  little  or  no 
financial  benefits 


What  are  the  underlying  principles  of  CMMI  as  they  relate  to 
productivity,  predictability,  and  speed? 

What  is  the  return  on  investment? 

What  are  the  timelines  for  realizing  these  benefits? 


CMMI®  is  a  registered  trademark  of  Carnegie  Mellon  University 
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Agenda 


■  CMMI  principles 

■  Industry  data  on  return  on  investment 

■  A  framework  for  measuring  benefits 

■  Project  performance  benefits 

■  Organizational  performance  benefits 

■  Northrop  Grumman  experience 

■  Strategic  actions  needed  to  extract  value  from  maturity 
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Projects  Have  Historically  Suffered  from 
Mistakes 


People-Related  Mistakes 

1 .  Undermined  motivation 

2.  Weak  personnel 

3.  Uncontrolled  problem 
employees 

4.  Heroics 

5.  Adding  people  to  a  late 
project 

6.  Noisy,  crowded  offices 

7.  Friction  between  developers 
and  customers 

8.  Unrealistic  expectations 

9.  Lack  of  effective  project 
sponsorship 

10.  Lack  of  stakeholder  buy-in 

1 1 .  Lack  of  user  input 

12.  Politics  placed  over 
substance 

13.  Wishful  thinking 


Process-Related  Mistakes 

14.  Overly  optimistic  schedules 

15.  Insufficient  Risk 
Management 

16.  Contractor  failure  Insufficient 
planning 

17.  Abandonment  of  planning 
under  pressure 

18.  Wasted  time  during  the 
fuzzy  front  end 

1 9.  Shortchanged  upstream 
activities 

20.  Inadequate  design 

21 .  Shortchanged  quality 
assurance 

22.  Insufficient  management 
controls 

23.  Premature  or  too  frequent 
convergence 

25.  Omitting  necessary  tasks 
from  estimates 

26.  Planning  to  catch  up  later 

27.  Code-like-hell  programming 


Product-Related  Mistakes 

28.  Requirements  gold-plating 

29.  Feature  creep 

30.  Developer  gold-plating 

31 .  Push  me,  pull  me 
negotiation 

32.  Research-oriented 
development 

Technology-Related  Mistakes 

33.  Silver-bullet  syndrome 

34.  Overestimated  savings  from 
new  tools  or  methods 

35.  Switching  tools  in  the  middle 
of  a  project 

36.  Lack  of  automated 
source-code  control 


Standish  Group,  2003 
survey  of  13,000  projects 

•  34%  successes 

•  15%  failures 

•  51%  overruns 


Reference:  Steve  McConnell,  Rapid  Development 
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Many  Approaches  to  Solving  the  Problem 


■  Which  weaknesses  are  causing  my  problems? 

■  Which  strengths  may  mitigate  my  problems? 

■  Which  improvement  investments  offer  the  best  return? 


People 
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Approaches  to  Process  Improvement 


Data-Driven  (e.g.,  Six  Sigma ,  Lean) 


t  t  t  t  t 

-  Clarify  what  your  customer 
wants  (Voice  of  Customer) 

■  Critical  to  Quality  (CTQs) 

■  Determine  what  your  processes 
can  do  (Voice  of  Process) 

■  Statistical  Process  Control 

■  Identify  and  prioritize 
improvement  opportunities 

■  Causal  analysis  of  data 

■  Determine  where  your 
customers/competitors  are 
going  (Voice  of  Business) 

■  Design  for  Six  Sigma 


Model-Driven  (e.g.,  CMM,  CMMI) 


■  Determine  the  industry  best 
practice 

■  Benchmarking,  models 

■  Compare  your  current  practices 
to  the  model 

■  Appraisal,  education 

■  Identify  and  prioritize 
improvement  opportunities 

■  Implementation 

■  Institutionalization 

-  Look  for  ways  to  optimize  the 
processes 
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What  is  the  CMM? 


■  Capability  Maturity  Models  are  a  structured  set  of  industry  best 
practices 

■  Based  on  industry  research  and  expert  consensus 

■  People  adopt  CMMs  to  emulate  the  behavior  (and  hopefully, 
performance)  of  successful  organizations 

■  The  value  of  a  CMM  is  dependent  upon 

■  Understanding  the  new  practices  you  are  adopting 

■  Adapting  them  to  your  environment 

■  Staying  with  them  long  enough  to  see  the  benefits 
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How  Do  Mature  Processes  Help? 


Process  maturity  gets  at  one 
source  of  the  problem,  e.g., 

■  Are  we  using  proven 
industry  practices? 

■  Does  the  staff  have  the 
resources  needed  to 
execute  the  process? 

■  Is  the  organization  providing 
effective  project  support? 

The  main  benefits  typically 
seen  are: 

■  Improved  predictability  of 
project  budgets  and 
schedules 

■  Improved  management 
awareness  of  problems 

■  Reduced  re-work,  which 
improves  predictability,  cost, 

and  schedule 


J.  Herbsleb  and  D.  Zubrow, 
“Software  Process  Improvement: 
An  Analysis  of  Assessment  Data 
and  Outcomes” 

■  13  organizations 

■  ROI  of  4:1  to  9:1 

■  Improved  quality,  error  rates, 
time  to  market,  productivity 


R.  Dion,  “Process  Improvement 
and  the  Corporate  Balance  Sheet” 

■  ROI  of  7.7:1 :  Reduced  re-work, 
improved  quality 

■  Two-fold  increase  in 
productivity 
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The  Knox  Cost  of  Quality  Model 


"  Extension  of  the  Cost  of  Quality  model  used  in  manufacturing 


Cost 

Category 

Definition 

Typical  Costs  for  Software 

Conformance 

Appraisal 

Discovering  the 
condition  of  the 
product 

Testing  and  associated 
activities,  product  quality 
audits 

Prevention 

Efforts  to 
ensure  product 
quality 

SQA  administration, 
inspections,  process 
improvements,  metrics 
collection  and  analysis 

Non¬ 

conformance 

Internal 

failures 

Quality  failures 
detected  prior 
to  product 
shipment 

Defect  management,  rework, 
retesting 

External 

failures 

Quality  failures 
detected  after 
product 
shipment 

Technical  support,  complaint 
investigation,  defect 
notification 

Knox’s  Theoretical  Model  for  Cost  of  Software  Quality  (Digital  Technical  Journal,  vol.5,  No.  4.,  Fall  1993,  Stephen  T.  Knox.) 
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Knox  Model  -  Theoretical  Benefits 


Prevention -a-  Appraisal  -b-  Int  Failure  Ext  Failure-*-  TCoSQ 


SEI  CMM  Level 
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Where  the  Problem  Sometimes  Arises 


■  Some  organizations  are  driven  to  achieve  a  maturity  level  only 
for  it’s  marketing  value 


Improvement  goals  are  not  set  ^ 

realistically  (“Level  5  in  ’05”)  , 

v  No  one  takes  the  improvement 

S  effort  seriously 

Only  some  of  the  projects  participate  ^ 
in  the  improvement  effort  ^ 

l  Personnel  perceive  CMM/CMMI 
r  as  more  expensive 

Only  some  of  the  projects  get  y 

appraised  m 

L  The  other  projects  don’t 
r  implement 

Insufficient  resources  (e.g.,  training,  ^ 
QA,  metrics,  consultants)  / 

l  People  don’t  learn  the  new 
?  behaviors  or  become  proficient 

Management  doesn’t  enforce  the  ^ 

process  ^ 

^  The  benefits  are  not  realized 
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CMMI  Attacks  Several  Dimensions  of  the 
Problem 
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Project  Performance 


CMMI 

■  Project  performance 
problems  often  arise 
because  of  incomplete  or 
unrealistic  planning 

■  Forgotten  activities 

■  Unconscious  decisions 

■  Overly-optimistic  estimates 

-  When  cost/schedule 
pressure  arises,  people 
abandon  the  plans,  leading 
to  more  problems 

■  Individual  judgment  versus 
best  use  of  resources 

-  Identifies  the  elements  of  good 
planning 

■  Proven  engineering  processes 

■  Estimates  based  on  historical 
data,  using  these  processes 

-  When  cost/schedule  pressure 
l  arises,  CMM/CMMI  practices 

)  track  and  correct 

■  Reactive  (L2) 

■  Proactive,  risk  management 
(L3) 

■  Quantitative  management  (L4) 

■  QA,  management  ensures 
processes/plans  are  followed 

T-  Train  project  managers  on  how  to  use  the  tools  (estimation, 
earned  value ,  risk  management) 

■  Project  managers  (not  organizational  staff)  must  be  responsible 
for  implementing  the  improved  processes 
■  Demand  realistic,  data-driven  estimates  Northrop  Grumman 
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Organizational  Performance 


CMMI 

-  Each  project’s  processes  are 
unique 

■  Personnel  must  re-learn 
with  each  project 

■  Difficulty  moving  people 
from  project  to  project 

■  Historical  data  of  little  use  in 
estimation 

■  No  way  to  compare  project- 
to-project 

■  Which  process  was  best? 

■  What  did  we  learn? 

-  Standard  organizational 
process,  tailored  to  fit  each 
project 

■  Can  be  documented,  trained, 
supported  by  templates 

■  Over  time,  people  learn  the 
process 

)■  Common  processes/measures 
allow  better  use  of  historical 
data 

■  Calibrate  cost  estimation 
models 

■  Project  to  project  comparisons 

■  Over  time,  the  organization  can 
optimize  the  process 

T-  Develop  an  organizational  process(es)  which  fits  the  full  range  of 
your  projects  (small/large,  all  life  cycles  and  project  types) 

■  Capture  and  use  historical  data  (measurement  repository) 

■  Capture  and  share  project  documents  (process  asset  library) 
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Rework/ Quality 


Focus  on  “faster  and 
cheaper”  leads  to  skipping  of 
essential  steps 

■  Key  steps  are  not  obvious, 
often  counter  intuitive 

Fixing  latent  defects  often 
accounts  for  30-40%  of 
project  cost 

■  The  cost  of  defects  (rework) 
is  seldom  measured 


CMMI 


A  disciplined  engineering  and 
management  process 

■  Do  it  right  the  first  time 

■  CM  M/CM  Ml  identifies  the 
essential  steps 

Peer  reviews  find  defects  early, 
where  it  is  cost  effective  to  fix 
them 

■  Requirements,  designs,  code, 
plans,  etc. 

■  Often  more  efficient  and 
effective  than  testing 

■  Many  types  (Fagan 
inspections,  walkthroughs, 
desk  checks,  etc.) 


Focus  on  eliminating  defects,  not  on  faster  and  cheaper 
Measure  the  cost  of  finding  and  fixing  defects 
Invest  time  in  learning  different  methods  of  peer  review  and  when 
each  is  effective 
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Institutionalization 


CMMI 


Some  improvement  efforts 
focus  on  quick  fixes 

■  Driven  by  yearly  budget 
cycles 

■  Expectation  that  results  will 
be  immediate 

It  is  tempting  to  reduce 
overhead  to  reduce  cost 

■  Training 

■  Staff  support  to  projects 

■  Use  of  outside  process 
experts 


Short-term  investment  for  long¬ 
term  gain 

■  Initial  investment  in  the  cost  of 
change,  learning  curve,  new 
overhead  structures 

■  Long-term  benefits  in  increased 
productivity 

Organizational  infrastructure 
exists  to  support  the  policies 
and  process 

■  Measurement  repositories 


Expect  18-24  months  before  benefits  begin  to  be  realized 
Senior  management  must  demand  that  everyone  follow  the  new 
processes 

QA  can  be  the  organization’s  strongest  tool  -  if  they  are  focused! 
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Benefits 


The  typical  benefits  are: 

■  Reduced  cost 

■  Faster  schedules 

■  Greater  productivity 

■  Higher  quality 


COSTS 

*  Investments 

•  Expenses 


BENEFITS 
•  Cost 


•  Schedule 

•  Quality 


*  Customer 
Satisfaction 


ROI  &  ic 


Cost -Benefit 


■  Increased  customer  satisfaction 

Over  40  published  studies  on  the  benefits  of  CMM 

■  DoD  DACS  website:  http://www.thedacs.com/databases/roi/ 

Similar  results  starting  to  be  seen  for  CMMI 

■  “Demonstrating  the  Impact  and  Benefits  of  CMMI:  An  Update 
and  Preliminary  Results,”  Software  Engineering  Institute, 
CMU/SEI-2003-SR-009,  Oct  2003 

■  http://www.sei.cmu.edu/cmmi/results/results-by-category.html 
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Typical  CM  MI  Benefits  Cited  in  Literature 


Reduced  Costs 

■  33%  decrease  in  the  average 
cost  to  fix  a  defect  (Boeing) 

■  20%  reduction  in  unit  software 
costs  (Lockheed  Martin) 

■  Reduced  cost  of  poor  quality 
from  over  45  percent  to 
under  30  percent  over  a 
three  year  period  (Siemens) 

■  1 0%  decrease  in  overall  cost 
per  maturity  level  (Northrop 
Grumman) 

Faster  Schedules 

■  50%  reduction  in  release 
turnaround  time  (Boeing) 

■  60%  reduction  in  re-work 
following  test  (Boeing) 

■  Increase  from  50%  to  95%  the 
number  of  milestones  met 
(General  Motors) 


■  Greater  Productivity 

■  25-30%  increase  in 
productivity  within  3  years 
(Lockheed  Martin,  Harris, 
Siemens) 

■  Higher  Quality 

■  50%  reduction  of  software 
defects  (Lockheed  Martin) 

■  Customer  Satisfaction 

■  55%  increase  in  award  fees 
(Lockheed  Martin) 
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Projaci 

Kirvi|imtnt 


Business 

AtquEslLian 


Policies,  Processes, 
Templates  &  Tools 


Organizational  Infrastructure  Required  for 
CM  MI  Level  3 


Process  Group 


Training  Program 


Process  Improvement 


Six  Sigma  Projects 

DEFINE  MEASURE  ANALYZE 


JE( 


Audits  &  Appraisals 


Communications 


Measurement  Repositories 
Predictive  Modeling 


m 


Best-Practice  Libraries 


Developing  and  maintaining  mature  processes  requires 
significant  time  and  investment  in  infrastructure 


NORTHROP  GRUMMAN 


Northrop  Grumman  Mission  Systems  Approach 


Mission  Success  Requires  Multiple  Approaches 


Risk  Management 


Dashboards  for  Enterprise- 
Wide  Measurement 


Systems  Engineering 


Communications  & 
Best-Practice  Sharing 


Independent  Reviews 


a®  D-bust  Governance  Model 
Policies,  Processes, 


CMMI  Level  5  for  Software, 
Systems,  and  Services 

ISO  9001  and  AS-9100 
Certification 


Six  Sigma 
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Process  Effectiveness 


Audits  &  Appraisals 


Staff  Competence  &  Training 


Decision  Analysis  & 

Resolution  ^  >-2aL: 


Requirements 

Development 

100.0X  T 


^Technical  Solution 


Rsk  Management  a 


Integrated  Project 
Management 


Organizational  Training^ 


Level 2 -Managed 


Requirements  Management 


SGI 


SP1.1 


Requirements  are  managed  and  inconsistencies  wi  project  plans  and 
ideried. 


Does  lie  project  develop  an  understanding  vi\  lie  requirements 
providers  on  lie  meaning  of  lie  requirements? 


flnoc  frio  nmiort  nhfain  mmmitnnonHn  bn  rom  liromonte  irnm  fa 


meeting' 
records,  an  a 
written  requirements 


20  Northrop  Grumman  sites 
externally  appraised  at  CMMI  Level  5 


Process  Asset  Library 


Process  Assets  Library 


Policy  ( 

Policy  Sort 

E  701  Quality  Managenl 
+  702  Documents  8e  Re 
E  911  Org  Process  Defi 
E  912  Org  Process  Foe 
E  913  Org  Process  Perl 
E  914  Org  Innovation  £ 
+  915  Org  Training 
E  921  Project  Planning 

□  H  921  Project  Plar 

□  ]  921  Project  Plar 

□  Hi  PP  Process  Audi 

□  H  Training:  921. 

□  Hi  Training:  921. 

□  H  Training:  921, 

E  Checklists 

Ei  Integrated  Mast 
0  Integrated  Mgm 


□  mi-OTT  Aroz.  I  rin 


NOR 

— 

1  C'  =1  f  1=1  i“i  i-  r 1 

nsr 


CMMI  &  Six  Sigma  courses 
Policies  &  processes  course 
Standard  Training  Modules 
for  each  job  function:  engineering, 
project  management,  QA,  CM,  etc. 


Communications  &  Collaboration 


NORTHROP  GRUMMAN 


Mission  Systems 


CMMI/Process  Mgmt 


Home  »  CMMI  /  Process  Management 


PI  Summit 


Si 


Mission  Systems 

About  Northrop  Grumman 

» 

News 

» 

Services 

» 

Programs  &  Initiatives 

» 

Welcome 


This  web  site  is  designed  to  support  the  sharing  of 
information  and  assets  related  to  Northrop  Grumman 
Mission  Systems'  Capability  Maturity  Model  Integrated 
(CMMI)  initiative. 


Assuring  mission  success  by  making  the  people 
and  processes  more  informed  and  effective 
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Program  Effectiveness 


Six  Sigma  connects  process  improvement  and  business  value 


DEFINE 


MEASURE 


ANALYZE 


IMPROVE 


CONTROL 


Charter  team,  \ 
map  process  &  y 

specify  CTQs  y' 

Measure  process  N. 
performance  y 

Identify  & 

quantify  root  y 

causes 

Select,  design 
implement  y 

solution 

Institutionalize 
improvement, 
ongoing  control 


"  Six  Sigma  projects  can  help  focus  and  measure  CMMI-driven  process 
improvements 

■  Identify  the  customer’s  needs,  maximize  the  value/cost 

■  Tools  for  management  by  variation  (CMMI  Levels  4  and  5) 

-  Results  to  date 

■  4000  Green  Belts,  200  Black  Belts,  12  Master  Black  Belts 

■  500  completed  Six  Sigma  projects,  250  in  progress 

■  Significant  benefit  to  our  customer  -  lower  costs,  better  performance 


Assuring  mission  success  by  identifying  the 
customer's  needs  and  reducing  defects 
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Operational  Effectiveness 
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Lessons  Learned 


■  Process  improvement  means  changing  the  process 

■  More  important  to  learn  the  new  behaviors  than  to  “go  through 
the  motions” 

■  Resistance  often  comes  from  fear  of  failure 

■  Walk  the  talk  --  management  at  all  levels  must  communicate  the 
need  for  continuous  improvement 

■  Focus  on  learning  from  your  mistakes  and  getting  better 

■  Training  and  assistance  helps  people  in  trying  new  processes 

■  Six  Sigma  is  a  strong  enabler  for  process  improvement 

■  Focus  on  data,  measurement  systems,  process  improvement 

■  Tying  improvements  to  business  goals 

■  Allows  the  projects  and  organization  to  optimize  the  CMMI 
practices  for  maximum  customer  benefit 
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NORTHROP  GRUMMAN 

Defining  the  future 

Service 
Extensions  to 
the  CMMI 

5th  Annual  CMMI  Technology  Conference 
Denver,  CO 

November  14-17,  2005 


Craig  R.  Hollenbach 

Technical  Fellow 

Northrop  Grumman  Corporation 


Agenda 


■  Purpose 

■  Sponsors  &  Membership 

■  Service  Coverage  in  Existing  CMMI-SE/SW,  vl.1 

■  Industry  Service  Models  or  Standards 

■  Rationale  for  a  Services  CMMI 

■  Schedule 

■  Industry  Participation 

■  Issues  References 
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Purpose  &  Sponsors 


■  Purpose: 

■  to  extend  the  CMMI  framework  to  cover  the 
provision  of  services 

■  Sponsors: 

■  CMMI  Steering  Group 

■  DoD  OSD 

■  NDIA,  Systems  Engineering  Division 

■  SEI 

■  Northrop  Grumman  -  proposed  to  sponsor  a 
Services  CMMI  to  the  CMMI  Steering  Group  in  Nov 
2004 
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Services  Team  Membership 


t  o  MAH 


Membership: 

Jeff  Zeidler  (Boeing) 

Steve  Stern  (Lockheed  Martin) 

Brandon  Buteau  (Northrop  Grumman) 

Craig  Hollenbach  (Northrop  Grumman)  -  Lead 
Roy  Porter  (Northrop  Grumman) 

Hal  Wilson  (Northrop  Grumman) 

Gordon  Ward  (Raytheon) 

Jerry  Simpson  (SAIC) 

Drew  Allison  (SSCI) 

Eileen  Forrester  (SEI) 

Barbara  Tyson  (SEI) 

Eileen  Clark  (SR A)  — 


{  vU  SH'iJnY  Vlrlhin 

Software  Engineering  Institute 


T7^ 


NORTHROP  GRUMMAN 


Raytheon 


An  Employes- Owned  Company 


SYSTEMS  AND 
SOFTWARE 
CONSORTIUM 


Service  Coverage  in  CMMI-SE/SW,  vl.l 


Vl.l  Content 
Services  CMMI  Content 
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What  is  Not  Covered  in  CMMI-SE/SW,  vl.l 


Candidate  model  content  could  cover: 

■  Service  Request  and  Incident  Management 

■  Service  requests  and  incidents  regarding  the  service  are  identified, 
registered,  tracked,  analyzed,  and  resolved. 

■  Capacity  Management 

■  Responsible  for  ensuring  adequate  capacity  is  available  at  all  times 
to  meet  the  requirements  of  the  business. 

■  Availability  Management 

■  Process  of  managing  the  ability  of  a  component  or  service  to 
perform  its  required  function  at  a  stated  instant  or  over  a  stated 
period  of  time 

■  Service  Continuity  Management 

■  Concerned  with  managing  an  organization's  ability  to  continue  to 
provide  a  pre-determined  and  agreed  level  of  IT  services  to  support 
the  minimum  business  requirements  following  an  interruption  to  the 
business. 
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What  is  Not  Covered  in  CMMI-SE/SW,  vl.l 
(cont.) 


■  Release  Management 

■  Process  of  testing  and  introducing  together  a  collection  of 
new  and/or  changed  configuration  items  into  the  live 
environment 

■  Service  Delivery 

■  Consistently  perform  a  well-defined  service  delivery  process 
that  integrates  all  service  delivery  activities  to  deliver  correct, 
consistent  IT  services  effectively  and  efficiently. 

■  Resource  Management 

■  Control  of  the  resources  (hardware  and  software)  needed  to 
deliver  the  services  is  maintained. 
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Industry  Service  Models  or  Standards 


Candidate  IT  service  models  and  standards  include: 

■  Information  Technology  Infrastructure  Library  (ITIL) 

■  Control  Objects  for  Information  and  related 
Technology  (COBIT) 

■  Information  Technology  Services  CMM  (ITSCMM) 

■  British  Standard  15000:  IT  Service  Management  (BS 
1 5000) 

The  Services  CMMI  team  is  investigating  non-IT 
service  models  and  standards. 
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Rationale  -  If  IT  service  models  exist, 
why  do  we  need  a  CM  MI  for  Services? 


■  The  CMMI  emphasizes  institutionalization  of 
process  maturity. 

■  The  CMMI  divides  improvements  into  incremental  efforts. 

■  A  CMMI  for  Services  would  rapidly  leverage 
investments  by  the  current  CMMI  user  base  to  bring 
process  maturity  to  their  services  efforts. 

■  CMMI-based  improvements  have  a  demonstrated  ROI. 

■  The  CMMI  provides  a  familiar  vocabulary. 

■  There  is  little  guidance  for  appraisers  and 
organizations  on  applying  the  CMMI  to  services 
efforts. 

■  “Implementation  models”  within  companies  differ  between 
SE/SW  and  services. 

■  The  CMMI  is  supported  by  standard  appraisal  methods. 
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Rationale  -  If  IT  service  models  exist, 
why  do  we  need  a  CM  MI  for  Services? 


■  Current  IT  models  do  not  address  the  development 
of  service  systems  as  thoroughly  as  the  CMMI. 

■  A  CMMI  for  Services  would  summarize  essential 
elements  from  current  IT  service  models. 

■  Maps  from  IT  service  models  to  a  Services  CMMI 
would  enable  organizations  to  refer  to  existing 
models  for  extensive  best  practices  for  services 

■  Reduces  preparation  costs  for  appraisals  against 
multiple  models 
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Why  do  we  need  a  CMMI  for  Services? 


■  Applying  the  CMMI  to  services  requires  significant 
interpretation  of  appraisers  and  organizations,  but 
there  is  no  guidance. 

■  Current  IT  services  models  do  not  completely 
address  service  development  and  initiation. 

■  A  CMMI  for  Services  would  re-use  a  familiar 
vocabulary  and  common  model  components. 

■  A  CMMI  for  Services  would  summarize  essential 
elements  from  current  IT  service  models. 

■  “Implementation  models”  for  development  differ 
from  those  for  services. 
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Why  do  we  need  a  CM  MI  for  Services? 


■  A  CMMI  for  Services  includes  additional  process 
areas  necessary  for  full  process  institutionalization 
and  innovation. 

■  CMMI  for  Services  maturity  levels  divide 
improvements  into  incremental  efforts. 

■  An  extensive  CMMI  user  community  can  leverage 
the  CMMI  framework  to  extend  current  maturity  into 
service  domains. 

■  A  CMMI  for  Services  would  summarize  essential 
principles  from  and  provide  maps  to  current  IT 
service  models,  allowing  for  integrated  improvement 
efforts  and  coordinated  best  practices. 
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Completed 
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Incremental  Pilots 
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Nov-06 


Industry  Participation 


■  Participate  in  piloting  of  Services  CMMI 

■  Provide  model  feedback 

■  Visit  the  public  Services  CMMI  (BSCW  site) 
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Accomplishments  To  Date 


■  The  Services  CMMI  Team  has  coordinated  well  with  the 
version  1 .2  team  on  saving  the  CMMI  framework’s 
common  content. 

■  The  Services  CMMI  Team  has  identified  few  changes  or 
impacts  in  the  generic  content. 

■  Lots  of  interest  from  across  the  world  indicates  that 
there  is  value  concentrating  on  Services  as  a 
Constellation. 

■  The  Services  CMMI  Team  has  identified  to  the  version 

1 .2  team  several  “thorny  issues”  that  require  interaction 
with  other  CMMI  author  teams. 
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What  is  the  scope  of  the  Services  CM  MI? 


■  Processes  would  include  both 

■  Development  of  systems  for  delivery  of  services 
(manual  and  automated)  -  applying  the  existing 
engineering  PA’s  -  and 

■  Delivery  of  services 

■  Service  Domains  would  include 

■  Professional  services  (e.g.,  engineering  services, 
technical  support,  military  equipment  maintenance, 
resupply  services)  typically  outside  the  domain  of  IT 
services 

■  Focus  on  IT  services,  but  broadly  defining  services  to 
not  *exclude*  other  industries 

■  Concern:  Some  industries  (e.g.,  medical,  utilities) 
may  have  specialized  service  models  that  may 
require  additional  coordination  and  research 
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"Thorny"  CMMI  Terminology 


■  The  Services  CMMI  team  uncovered  the  following 
‘thorny’  terminology  issues. 

■  Work  Product  (the  current  definition  includes 
services) 

■  Product  (called  tangible) 

■  Project  (definite  beginning  and  end) 

■  Management  and  Technical  Roles  (organizational 
training  definition) 

■  Product  Quality  (not  consistently  applied) 
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Challenges 


■  Discern  the  process  areas  and  practices  that  are 
common  to  a  wide  variety  of  services 

■  Select  model  language  that  communicates  to  the 
service  industry  but  causes  as  small  an  impact  as 
possible  to  existing  model  content 

■  Maximize  coordination  with  existing  services 
organizations 

■  Manage  the  size  and  complexity  of  the  services  model 

■  Effectively  coordinate  common  CMMI  content  with  the 
development  constellation  (vl  .2) 

■  Support  organizations  that  perform  both  development 
and  service  delivery. 
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Purpose  &  Sponsors 


■  Purpose: 

■  to  extend  the  CMMI  framework  to  cover  the 
provision  of  services 

■  Sponsors: 

■  CMMI  Steering  Group 

■  DoD  OSD 

■  NDIA,  Systems  Engineering  Division 

■  SEI 

■  Northrop  Grumman  -  proposed  to  sponsor  a 
Services  CMMI  to  the  CMMI  Steering  Group  in  Nov 
2004 


NORTHROP  GRUMMAN 


CMMI  Technology  Conference,  November  14-17,  2005 


Services  Team  Membership 


t  o  MAH 


Membership: 

Jeff  Zeidler  (Boeing) 

Steve  Stern  (Lockheed  Martin) 

Brandon  Buteau  (Northrop  Grumman) 

Craig  Hollenbach  (Northrop  Grumman)  -  Lead 
Roy  Porter  (Northrop  Grumman) 

Hal  Wilson  (Northrop  Grumman) 

Gordon  Ward  (Raytheon) 

Jerry  Simpson  (SAIC) 

Drew  Allison  (SSCI) 

Eileen  Forrester  (SEI) 

Barbara  Tyson  (SEI) 

Eileen  Clark  (SR A)  — 
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Service  Coverage  in  CMMI-SE/SW,  vl.l 


Vl.l  Content 
Services  CMMI  Content 
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What  is  Not  Covered  in  CMMI-SE/SW,  vl.l 


Candidate  model  content  could  cover: 

■  Service  Request  and  Incident  Management 

■  Service  requests  and  incidents  regarding  the  service  are  identified, 
registered,  tracked,  analyzed,  and  resolved. 

■  Capacity  Management 

■  Responsible  for  ensuring  adequate  capacity  is  available  at  all  times 
to  meet  the  requirements  of  the  business. 

■  Availability  Management 

■  Process  of  managing  the  ability  of  a  component  or  service  to 
perform  its  required  function  at  a  stated  instant  or  over  a  stated 
period  of  time 

■  Service  Continuity  Management 

■  Concerned  with  managing  an  organization's  ability  to  continue  to 
provide  a  pre-determined  and  agreed  level  of  IT  services  to  support 
the  minimum  business  requirements  following  an  interruption  to  the 
business. 
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What  is  Not  Covered  in  CMMI-SE/SW,  vl.l 
(cont.) 


■  Release  Management 

■  Process  of  testing  and  introducing  together  a  collection  of 
new  and/or  changed  configuration  items  into  the  live 
environment 

■  Service  Delivery 

■  Consistently  perform  a  well-defined  service  delivery  process 
that  integrates  all  service  delivery  activities  to  deliver  correct, 
consistent  IT  services  effectively  and  efficiently. 

■  Resource  Management 

■  Control  of  the  resources  (hardware  and  software)  needed  to 
deliver  the  services  is  maintained. 
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Industry  Service  Models  or  Standards 


Candidate  IT  service  models  and  standards  include: 

■  Information  Technology  Infrastructure  Library  (ITIL) 

■  Control  Objects  for  Information  and  related 
Technology  (COBIT) 

■  Information  Technology  Services  CMM  (ITSCMM) 

■  British  Standard  15000:  IT  Service  Management  (BS 
1 5000) 

The  Services  CMMI  team  is  investigating  non-IT 
service  models  and  standards. 
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Rationale  -  If  IT  service  models  exist, 
why  do  we  need  a  CM  MI  for  Services? 


■  The  CMMI  emphasizes  institutionalization  of 
process  maturity. 

■  The  CMMI  divides  improvements  into  incremental  efforts. 

■  A  CMMI  for  Services  would  rapidly  leverage 
investments  by  the  current  CMMI  user  base  to  bring 
process  maturity  to  their  services  efforts. 

■  CMMI-based  improvements  have  a  demonstrated  ROI. 

■  The  CMMI  provides  a  familiar  vocabulary. 

■  There  is  little  guidance  for  appraisers  and 
organizations  on  applying  the  CMMI  to  services 
efforts. 

■  “Implementation  models”  within  companies  differ  between 
SE/SW  and  services. 

■  The  CMMI  is  supported  by  standard  appraisal  methods. 
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Rationale  -  If  IT  service  models  exist, 
why  do  we  need  a  CM  MI  for  Services? 


■  Current  IT  models  do  not  address  the  development 
of  service  systems  as  thoroughly  as  the  CMMI. 

■  A  CMMI  for  Services  would  summarize  essential 
elements  from  current  IT  service  models. 

■  Maps  from  IT  service  models  to  a  Services  CMMI 
would  enable  organizations  to  refer  to  existing 
models  for  extensive  best  practices  for  services 

■  Reduces  preparation  costs  for  appraisals  against 
multiple  models 
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Why  do  we  need  a  CMMI  for  Services? 


■  Applying  the  CMMI  to  services  requires  significant 
interpretation  of  appraisers  and  organizations,  but 
there  is  no  guidance. 

■  Current  IT  services  models  do  not  completely 
address  service  development  and  initiation. 

■  A  CMMI  for  Services  would  re-use  a  familiar 
vocabulary  and  common  model  components. 

■  A  CMMI  for  Services  would  summarize  essential 
elements  from  current  IT  service  models. 

■  “Implementation  models”  for  development  differ 
from  those  for  services. 
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Why  do  we  need  a  CM  MI  for  Services? 


■  A  CMMI  for  Services  includes  additional  process 
areas  necessary  for  full  process  institutionalization 
and  innovation. 

■  CMMI  for  Services  maturity  levels  divide 
improvements  into  incremental  efforts. 

■  An  extensive  CMMI  user  community  can  leverage 
the  CMMI  framework  to  extend  current  maturity  into 
service  domains. 

■  A  CMMI  for  Services  would  summarize  essential 
principles  from  and  provide  maps  to  current  IT 
service  models,  allowing  for  integrated  improvement 
efforts  and  coordinated  best  practices. 
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Package  and  release  draft  CMMI  for  Services  for  comment 
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Pilot  draft  CMMI  for  Services 

i  . 

Revise  draft  CMMI  for  Services 
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Package  and  release  final  CMMI  for  Services  j 

Completed 
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Incremental  Pilots 
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Nov-06 


Industry  Participation 


■  Participate  in  piloting  of  Services  CMMI 

■  Provide  model  feedback 

■  Visit  the  public  Services  CMMI  (BSCW  site) 
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Accomplishments  To  Date 


■  The  Services  CMMI  Team  has  coordinated  well  with  the 
version  1 .2  team  on  saving  the  CMMI  framework’s 
common  content. 

■  The  Services  CMMI  Team  has  identified  few  changes  or 
impacts  in  the  generic  content. 

■  Lots  of  interest  from  across  the  world  indicates  that 
there  is  value  concentrating  on  Services  as  a 
Constellation. 

■  The  Services  CMMI  Team  has  identified  to  the  version 

1 .2  team  several  “thorny  issues”  that  require  interaction 
with  other  CMMI  author  teams. 
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What  is  the  scope  of  the  Services  CM  MI? 


■  Processes  would  include  both 

■  Development  of  systems  for  delivery  of  services 
(manual  and  automated)  -  applying  the  existing 
engineering  PA’s  -  and 

■  Delivery  of  services 

■  Service  Domains  would  include 

■  Professional  services  (e.g.,  engineering  services, 
technical  support,  military  equipment  maintenance, 
resupply  services)  typically  outside  the  domain  of  IT 
services 

■  Focus  on  IT  services,  but  broadly  defining  services  to 
not  *exclude*  other  industries 

■  Concern:  Some  industries  (e.g.,  medical,  utilities) 
may  have  specialized  service  models  that  may 
require  additional  coordination  and  research 
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"Thorny"  CMMI  Terminology 


■  The  Services  CMMI  team  uncovered  the  following 
‘thorny’  terminology  issues. 

■  Work  Product  (the  current  definition  includes 
services) 

■  Product  (called  tangible) 

■  Project  (definite  beginning  and  end) 

■  Management  and  Technical  Roles  (organizational 
training  definition) 

■  Product  Quality  (not  consistently  applied) 
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Challenges 


■  Discern  the  process  areas  and  practices  that  are 
common  to  a  wide  variety  of  services 

■  Select  model  language  that  communicates  to  the 
service  industry  but  causes  as  small  an  impact  as 
possible  to  existing  model  content 

■  Maximize  coordination  with  existing  services 
organizations 

■  Manage  the  size  and  complexity  of  the  services  model 

■  Effectively  coordinate  common  CMMI  content  with  the 
development  constellation  (vl  .2) 

■  Support  organizations  that  perform  both  development 
and  service  delivery. 
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Reducing  Variation  at  Each 
CMMI  Maturity  Level 
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Continuous  Variation 
of  Reducing  Variation 


♦The  ideas  of  variation  found  in  this  presentation 

^Provide  the  backdrop  for  using  the  CMMI  model  as 
the  basis  for  an  organization’s  process  improvement 
initiative 

^Show  that  this  journey  is  synonymous  with  a  journey 
of  reducing  variation 
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Understanding 

Variation 


Understanding  Variation 
The  Key  to  Managing  Chaos 
Donald  J.  Wheeler,  SPC  Press,  2000 
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Process  Change  and 
Variation 


♦Dr.  Wheeler  shared  his  interpretation  of  Dr. 
Walter  Shewhart’s  approach  to  interpreting 
data. 

^“We  analyze  numbers  in  order  to  know  when  a 
change  has  occurred  in  our  process  of  system....” 

■  ■  ■ 

^Some  variation  is  routine,  run-of-the-mill,  and  is  to 
be  expected  even  when  the  process  has  not 
changed. 

mother  variation  is  exceptional,  outside  the  bounds  of 
routine,  and  therefore  to  be  interpreted  as  a  signal  of 
process  change....”  "... 
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Understanding 
Variation 

♦  Understanding  variation  is  achieved  by  collecting 
and  analyzing  process  and  product  measures  so 
that  special  causes  of  variation  can  be  identified 
and  addressed  to  achieve  predictable  performance 

♦All  characteristics  of  processes  and  products 
display  variation  when  measured  over  time 

♦Variation  may  be  due  to 
<§>Natural  or  common  causes 
<$>Special  or  “assignable”  causes  of  variation 

♦  Understanding  and  controlling  variation  is  the 
essence  of  CMMI  Maturity  L4  &  L5 
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Common  Causes  of 
Variation 


♦Common  causes  of  variation 

^Variation  in  process  performance  due  to  normal 
interaction  among  the  process  components  (people, 
machines,  material,  environment,  and  methods) 

^Characterized  by  a  stable  and  consistent  pattern  of 
measured  values  over  time 

^Variation  due  to  common  cause  is  random  but  will 
vary  within  predictable  bounds 

^Unexpected  results  are  extremely  rare 

^Predictable  is  synonymous  with  in  control 
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Special  Causes  of 
Variation 


♦Special  or  Assignable  causes  of  variation 

Arise  from  events  that  are  not  part  of  the  normal 
process 

<$>  Represent  sudden  or  persistent  abnormal  changes 
due  to  one  or  more  of  the  process  components 

♦  inputs  to  the  process 

♦  environment 

♦  process  steps  themselves 

♦  the  way  the  process  steps  are  executed 

<§> Examples  of  assignable  causes  of  variation  include 
inadequately  trained  people,  tool  failures,  failures  to 
follow  the  process 
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Process  Variation 


♦  Reducing  process  variation  is  an  important 
aspect  to  quantitative  management: 

<§>lt  is  important  to  focus  on  subprocesses  that  can 
be  controlled  to  achieve  a  predictable 
performance 


♦Statistical  process  control  is  often  better 
focused  on  organizational  areas  such  as 
Product  Lines  where  there  is  high  similarity  of 
processes,  than  on  the  organization’s  entire  set 
of  products 
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CMMI  Overview 


Process  Areas 

Optimizing 

Focus  is  on  quantitative 
continuous  process 
improvement 

Causal  Analysis  and  Resolution 

Organizational  Innovation  and  Deployment 

Quantitatively 

Managed 

Process  is  measured 
and  controlled 

Quantitative  Project  Management 

Organizational  Process  Performance 

Defined 

Process  is  characterized 
for  the  organization  and 
is  proactive 

Requirements  Development  Integrated  Project  Management 
Technical  Solution  Integrated  Teaming 

Product  Integration  Organizational  Environment 

Verification  For  Integration 

Validation  Integrated  Supplier  Management 

Organizational  Process  Focus  Risk  Management 

Organization  Process  Definition  Decision  Analysis  &  Resolution 
Organizational  Training 

Managed 

Process  is  characterized 
for  projects  and  is  often 
reactive 

Requirements  Management  Configuration  Management 

Project  Planning  Measurement  and  Analysis 

Project  Monitoring  and  Control 

Supplier  Agreement  Management 

Product  and  Process  Quality  Assurance 

Initial 

Process  is  unpredictable, 
poorly  controlled,  and 
reactive 

Maturity  Level  1:  Initial 


♦Processes  are  usually  ad  hoc  and  chaotic 

♦The  organization  usually  does  not  provide  a 
stable  environment 

♦Success  depends  on  the  competence  and 
heroics  of  the  people  in  the  organization  and 
not  on  the  use  of  proven  processes 

♦Maturity  level  1  organizations  are  characterized 
by  a  tendency  to  over  commit,  abandon 
processes  in  the  time  of  crisis,  and  not  be  able 
to  repeat  their  past  successes 
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Variation  Among 
Individuals 


♦One  of  the  traits  of  CMMI  Maturity  Level  1  is 
that  the  process  “belongs”  to  the  people.” 

<^lf  others  follow  a  process,  it  is  normally  due  to  the 
strong  personality  of  someone  on  the  project  who 
has  experienced  using  processes  in  another 
environment 

♦From  a  variation  point  of  view,  a  level  one 
organization  has  great  variation  based  on  its 
individual  employees  following  their  own 
process  paths.  This  is  why  maturity  level  one 
companies  depend  so  heavily  on  the  heroics  of 
its  people 
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Maturity  Level  2: 
Managed 

♦  Projects  ensure  that  requirements  are 
managed  and  that  processes  are  planned, 
performed,  measured,  and  controlled 

♦The  process  discipline  reflected  by  maturity 
level  2  helps  to  ensure  that  existing  practices 
are  retained  during  times  of  stress 

♦At  maturity  level  2,  requirements,  processes, 
work  products,  and  services  are  managed 

<^The  status  of  the  work  products  and  the  delivery  of 
services  are  visible  to  management  at  defined  points 

^The  work  products  and  services  satisfy  their 
specified  requirements,  standards,  and  objectives 
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Managing  the  Project 
Involves 

♦  Estimating  the  scope  and  work  that  needs  to  be 
performed 

♦  Developing  mechanisms  to  acquire  identified  products 

♦  Developing  a  project  plan 

♦  Getting  commitments  to  the  plan 

♦  Working  with  suppliers  to  acquire  identified  products 

♦  Monitoring  progress  against  the  plan 

♦  Identifying  and  analyzing  risks 

♦  Taking  action  to  address  significant  deviations  from  the 
plan 

♦  Taking  action  to  appropriately  mitigate  risks 
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Measurement  and 
Analysis  to  Support 
Projects 

♦Support  projects  includes  specifying  the 
objectives  of  measurement  and  analysis  such 
that  they  are  aligned  with  established 
information  needs  and  business  objectives 

<§> Defining  the  measures  to  be  used,  the  data 
collection  process,  the  storage  mechanisms,  the 
analysis  processes,  the  reporting  processes,  and 

the  feedback  processes 

^Providing  objective  results  that  can  be  used  in 
making  business  judgments  and  taking  appropriate 
corrective  actions 
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Basic  Measures 


♦Project  Management  Measures 

<^Size  and  complexity 
<§>  Effort  and  Cost 
^Schedule 

^Computer  Resources 
Data  Management 
^Knowledge  and  Skills 
^Stakeholder  Involvement 
^Technical  Performance 
^Commitments 
^Critical  Dependencies 
<§>Quality 
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More  Advanced 
Measures 

♦Earned  value 
♦Defect  density 
♦Peer  Review  Effectiveness 
♦Testing  Effectiveness 
♦Test  Coverage 

♦  Reliability 
♦Maintainability 

♦  Interoperability 
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Project’s  Processes  to 
Reduce  Variation 


♦At  CMMI  Maturity  Level  2,  processes  normally 
belong  to  the  project  and  are  enforced  by  the 
Project  Manager 

♦The  processes,  standards,  guidelines, 
checklists,  and  templates  are  enforced  for  all  of 
the  project  members  to  achieve  more 
uniformity  in  development  and  product  quality 

♦  Assuming  that  all  projects  follow  some  form  of 
process,  the  amount  of  variation  that  was  seen 
in  organizations  of  maturity  level  1  is  reduced 
even  if  all  of  the  projects  followed  a  different 
process 
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Maturity  Level  3 
Defined 


♦Processes  are  well  characterized  and 
understood,  and  are  described  in  standards, 
procedures,  tools,  and  methods 

♦The  organization’s  set  of  standard  processes, 
which  is  the  basis  for  maturity  level  3,  is 
established  and  improved  over  time. 

<^These  standard  processes  are  used  to  establish 
consistency  across  the  organization 

^Projects  establish  their  defined  processes  by 
tailoring  the  organization’s  set  of  standard 
processes  according  to  tailoring  guidelines 
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Maturity  Level  3: 
Defined  -  2 

♦The  organization’s  management  establishes 
process  objectives  based  on  the  organization’s 
set  of  standard  processes 

♦  Processes  are  typically  described  in  more 
detail  and  more  rigorously  than  at  maturity 
level  2 
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Organizational  Processes 
to  Reduce  Variation 

♦At  The  Organizational  Level,  an  organization 
that  wishes  to  achieve  CMMI  Maturity  Level  3 
needs  to  have  its  processes  owned  by  the 
organization  for  economy  of  scale  to  be 
realized  and  process  measurement  to  make 
practical  sense 

♦These  process  definitions  are  tailored  and 
incorporated  into  the  project’s  defined 
processes  throughout  the  organization  and 
thus  variation  in  project  development  and 
product  and  service  quality  is  again  reduced 
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Organizational  Processes 
to  Reduce  Variation  -  2 

♦An  organizational  measurement  repository  is 

established  and  maintained  which  contains 
both  product  and  process  measures  based  on 
the  organization’s  set  of  standard  processes 
along  with  the  information  needed  to 
understand  and  interpret  the  measures 

^Trends  can  be  seen  and  predictability  can  be  start  to 
be  achieved 

^Process  performance  baselines  can  now  be 
developed  to  support  quantitative  management  later 
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Organization’s  Process 


Measure! 
Repository 


Library 


Life-cycle 

Models 


Tailoring 

Guidelines 


Support 
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Maturity  Level  4: 
Quantitatively  Managed 


♦Quantitative  objectives  for  quality  and  process 
performance  are  established  and  used  as 
criteria  in  managing  processes. 

♦Quantitative  objectives  are  based  on  the  needs 
of  the  customer,  end  users,  organization,  and 
process  implementers 

^Quality  and  process  performance  are  understood  in 
statistical  terms  and  are  managed  throughout  the  life 
of  the  processes 

^Subprocesses  are  selected  that  significantly 
contribute  to  overall  process  performance 
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Maturity  Level  4: 
Quantitatively  Managed  -  2 


♦Special  causes  of  process  variation  are 
identified  and,  where  appropriate,  the  sources 
of  special  causes  are  corrected  to  prevent 
future  occurrences 

♦Quality  and  process  performance  measures 
are  incorporated  into  the  organization’s 
measurement  repository  to  support  fact-based 
decision  making  in  the  future 

♦The  performance  of  processes  is  controlled 
using  statistical  and  other  quantitative 
techniques 

<^At  maturity  level  3,  processes  are  only  qualitatively 
predictable. 
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Quantitative  Project 
Management 

♦Quantitative  Management  is  tied  to  the 
organization’s  strategic  goals  for  product  quality, 
service  quality,  and  process  performance 

♦When  higher  degrees  of  quality  and  performance 
are  demanded,  tne  organization  and  projects  must 
determine  if  they  have  the  ability  to  improve  the 
necessary  processes  to  satisfy  the  increased 
demands 

♦  Achieving  the  necessary  quality  and  process 
performance  objectives  requires  stabilizing  the 
processes  or  subprocesses  that  contribute  most  to 
the  achievement  of  the  objectives  and  reducing 
process  variation  to  support  the  quantitative 
management  objectives. 


©  2005  Kasse  I  initiatives,  LLC  Version  NDIA  CMMI  Conf  -  2005  Reducing  Variation  -  CMMi  -  26 


Process  and  Product 
Performance 

♦  Process  performance  is  a  measure  of  the  actual 
process  results  achieved 

♦  Process  performance  is  characterized  by  both 
process  measures  and  product  measures 

♦  Process  measures  include: 

•^Effort 
<§> Cycle  time 
Defect  removal  efficiency 

♦  Product  measures  include: 

•^Reliability 
<§>  Defect  density 
<§>  Response  time 
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Moving  from  Defined 
Processes  to  Quantitatively 
Managed  Processes 

♦With  defined  processes,  measures  are  collected 
and  analyzed  to  understand  and  manage  activities 
and  results: 

^Threshold  limits  are  set,  but  not  using  statistical  and 
other  quantitative  methods 

^Exceeding  threshold  limits  triggers  actions 

♦With  quantitative  management 

^Analyses  are  concerned  with  addressing  special  causes 
of  process  variation 

^Measurements  are  analyzed  quantitatively  to 

♦  Understand  process  performance 

♦  Predict  the  achievement  of  product  quality  and  service 
quality  objectives 
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Quantitative  Tools 


♦  There  are  a  number  of  quantitative  tools  considered  to 
be  applicable  to  statistical  process  or  quality  control: 

<§>  Quantifying  and  Predicting  Process  Performance 

♦  Control  Charts 

♦  Histograms 

♦  Run  charts 

<§>  Cause  and  Effect  Relationships 

♦  Scatter  diagrams 

♦  Cause-and-effect  (fishbone)  diagrams 

♦  Bar  charts 

♦  Pareto  charts 

♦  Interrelationship  Diagraph 

♦  Kiviat  Diagram 
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PROCESS  CONTROL  CHART  TYPE: 


METRIC: 


Upper 

Control 

Limit 

(UCL) 


f  A  point  above  or  below  the 
control  lines  suggests  that  the 
__  measurement  has  a  special 
preventable  or  removable  cause 


Upper  and 


Lower 
Control  Limits 
represent  the 
natural  variation 
In  the  process 


The  chart  is  used  for  continuous 
and  time  control  of  the  process 
and  prevention  of  causes 


Center  Line  (CL) 

(Mean  of  data  used  to 
set  up  the  chart) 

i  i 


Lower 

Control 

Limit 


Plotted  points  are  either 
individual  measurements  or  the 
means  of  small  groups  of 
measurements 


The  chart  is  analyzed  using 
standard  Rules  to  define  the 
control  status  of  the  process 


(LCL) 


Data 

relating  to 
the  process 


©  2005  Kasse  Initiate 


Numerical  data  taken 
in  time  sequence 


Statistical  Methods  for  Software  Quality 
Adrian  Burr  -  Mai  Owen,  1996 

KI  Additional  Slides:SEI  Intro  CMMI  Constagedeous 


30 


Number  of  Days 


Histogram 


Product-  Service  Staff  Hours 
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Quantitative  Project 
Management  Concepts 
References 

♦Some  sources  that  can  help  to  really 
understand  what  is  behind  this  statistical 
process  control  are: 

^Understanding  Variation  by  Donald  Wheeler 

^Statistical  Methods  for  Software  Quality  by  Adrian 
Burr  and  Mai  Owen 

^Measuring  the  Software  Process  by  William  Florae 
and  Anita  Carleton. 
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Voice  of  the 
Customer 
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Process  Capability 


♦Process  capability  is  analyzed  for  those 
subprocesses  and  those  measured  attributes 
for  which  objectives  have  been  set 

♦A  capable  process  is  one  that  is  satisfying  its 
quality  and  process  performance  objectives 
and  can  be  expected  to  satisfy  those  objectives 
in  the  future 
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Voice  of  the  Process 


♦Voice  of  the  Process 

^The  natural  bounds  and  variation  within  those 
bounds  of  process  performance 

♦  variation  is  within  3o  of  the  process  mean 

♦  process  is  stable  and  does  not  exhibit  any 
unlikely  patterns  or  events 
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Voice  of  the  Customer 


♦Voice  of  the  Customer 

<§>The  goals  established  for  the  product  and  process 
performance 

♦  product  specifications 

-  amount  of  downtime 

-  mean  time  to  failure 

-  response  time 

♦  management  specifications 

-  meeting  the  schedule 

-  meeting  the  budget 
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Capable  Processes 


Stable  Process 
(Process  Performance 
Is  Predictable) 


Quality  &  Process 
Performance 
Meets  Customer 
Requirements 


► 
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Summary 
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Process  Capability 
Prediction 


Focus  is  on  continuous 
quantitative  improvement 


Time/$/. 


Process  is  measured 
and  controlled 


Time/$/. 


Process  is  characterized 
for  the  organization  and 
is  proactive 


Process  is  characterized 
for  projects  and  is  often 
reactive 


Time/$/. 


Process  is  unpredictable, 
poorly  controlled,  and 
reactive 


Time/$/. 
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Summary 


♦  The  Process  Capability  Prediction  figure  provides  a  “process 
capability  prediction”  view  of  the  CIVtMl  and  illustrates  the  theme  of 
reduction  of  variation 

♦  Initial  level  -  target  dates  of  cost,  schedule,  performance  and 
quality  are  often  missed  by  wide  variation. 

♦  Managed  level  -  the  variability  of  the  actual  results  around  the 
target  decreases. 

♦  Defined  level  -  variability  again  decreases. 

<§>  Target  hits  increase  and  the 

<§>  Target  begins  to  move  in  toward  the  Y-axis  due  to  reduced  rework. 

♦  Quantitatively  Managed  level  -  variability  continues  to  decrease. 

<§>  Target  results  improve, 

^  Development  time  becomes  shorter 

^  Productivity  and  quality  increase. 

♦  Optimizing  level  -  defect  prevention  helps  to  reduce  rework 
further  andvariation  continues  to  be  reduced. 
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Summary  -  2 


♦There  are  also  many  different  ways  that  the 
CMMI  can  help  an  organization  that  are  not 
always  obvious  on  the  surface 

♦  Helping  an  organization  to  reduce  variation  as 
it  improves  in  its  process  capability  is  a  benefit 
of  using  the  CMMI  that  all  organizations  should 
strive  to  utilize 
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Kasse  Initiatives 
Contact  Information 

♦  Kasse  Initiatives  LLC 

<$>PMB293 

<$>1900  Preston  Road,  Suite  267 
<$>Plano,  Texas  75093 
<$>972-987-7601  Office 
<$>972  -  987  -  7607  FAX 
^www.kasseinitiatives.com 
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Sound  Systems  Engineering 
using  CMMI® 

Michael  T.  Kutch.  Jr.  Sandee  Guidry 

Director  Engineering  Operations,  Code  09K  SEI  Authorized  CMMI  Lead  Appraiser 

Chief  Engineer  Code  70E  SEI  Authorized  CMMI  Trainer 

Intelligence  &  Information  Warfare  Systems  Technical  Software  Services,  Inc. 

Department  Engineering  Process  Office 

SPAWAR  Systems  Center  Charleston  (SSC-C)  SPAWAR  Systems  Center  Charleston  (SSC-C) 

NDIA  CMMI  Technology  Conference,  November  17,  2005 
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What  We  Do 


SBBMBR 


Systems  Center 
Charleston 
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•  Modeling  &  Simulation 

•  Command  &  Control 

•  Navigation 

•  Physical  &  Computer 
Security 

•  Video  Teleconferencing 

•  Information  Assurance 

•  Sensors 

•  Communications 

•  Cryptologic  &  Intelligence 

•  Image  Processing 

•  Meteorology 

•  Air  Traffic  Control 
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What  We’re  Known  For 


Systems  Center 
Charleston 


•Developer  of  FORCEnet  joint  collaborative 
assessment  tools  that  promote  netCentric 
interoperability  and  reduce  system  redundancy 

•Principal  SPAWAR  provider  for  Joint  and 
Homeland  Security  C4I  solutions  in  a  responsive 
manner. 

•  Navy’s  most  efficient  provider  of  critical 
engineering  and  acquisition  expertise  for  Navy/Joint 
commands  and  other  federal  agencies 


•Rapid  integrator  and  deployer  of  interoperable  technologies  to  the 

Navy,  Federal  Government,  and  Joint  Warfighter 


•Developer  and  employer  of  life-cycle  logistic  support  solutions  in  a 

web-enabled  portal  environment 
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Who  We  Are 


Systems  Center 
Charleston 


A  Large  Systems  &  Software  Engineering  Organization 


Over  70%  of  workforce 
is  in  an  engineering  or 
computer-related 
discipline 


Computer 

Science/Engineering 
(185) 

Computer 
Specialist  (418) 


./Contracts  &  Supply  (122) 

•-Finance  &  Budget  (82) 

General  Clerical  (69) 

*1T  Support  (93) 

^Logistics  (73) 

Other  (170) 

Program  Management  (95) 


•The  effective  and  efficient  solutions  to  the  global  war  on  terror 
developed  by  SPAWAR  result  from  good  systems  and  software 
engineering. 

•Systems  engineering  is  our  core  competency. 


•Total  workforce  of  ~  2300  employees. 
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SE  Revitalization  Effort  using  CMMI® 


Systems  Center 
Charleston 


>  Vision 
>Organization 
>Plan 

>  Process 
>Tool 
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Systems  Center 
Charleston 


Vision 


•  Vision 

-  Develop  and  maintain  a  World  Class  Systems  Engineering  Organization 

•  Approach 

-  Achieve  Command-wide  operational  consistency 

-  Based  on  ISO/IEC  15288  -  systems  engineering 

-  Based  on  ISO/IEC  12207  -  software  engineering 

-  Based  on  implementing  CM  Ml®  “Staged  Respresentation” 

-  Measure  using  best  practices  of  CMMI®  “Continuous  Representation” 

•  Benefits 

-  Facilitates  sharing  of  tools,  documentation,  templates,  and  other  artifacts 
needed  by  project  engineers 

-  Project  Engineers  will  implement  projects  quicker;  with  improved 
monitoring,  effectiveness,  quality  and  efficiency 
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Organization  for  Implementation 


SBBMBR 


Systems  Center 
Charleston 
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SSC-C  SE  Revitalization  Plan 


SBBMBR 


Systems  Center 
Charleston 
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Basis  for  SSC-C  SE  Process 


Systems  Center 
Charleston 


Measure  &  Assess 
Processes 


CMMI® 

for 

SE/SW 


SPAWAR  Guidance 


Industry  Process 
Standards 


INCOSE 

GIG  /  Net-Centric  Strategy 

PMBOK/SWEBOK 

CJCS-JCIDS  3170.01 

IEEE 

DoD  Architecture 
Framework 

EIA  632 

SECNAV  5000.2C 

ISO  9001 :2000 

Quality  Systems 

DoD  5000.1/5000.2 

Industry  References  DoD  References 
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SSC-Charleston  SE  Process  Steps 


Systems  Center 
Charleston 


Process  Implementation 


Stakeholders  Requirements  Definition 


i 


Key  Milestones 

System  Requirements  Review  (SRR) 


System  Requirements  Analysis 


i 


System  Architectural  Design 


Implementation 


System  Design  Review  (SDR) 


Test  Readiness 
Review  (TRR) 


Integration 


Verification 


Transition 


Validation 


Each  process  step  is  defined  by  required  inputs, 
controls,  associated  processes,  and  outputs. 
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ePIan  Builder  Tool 


Systems  Center 
Charleston 


•  ePIan  Builder  tool 

-  An  interactive,  web-based  application  that  leads  the  user  through 
a  structured  interview  process  (like  TurboTax)  to  generate  a 
CMMI®-compliant  plan 

-  Includes  standard,  consistent  text 

-  Generates  a  complete  Project  Management  Plan,  Configuration 
Management  Plan,  Quality  Assurance  Plan,  and  Requirements 
Management  Plan 

-  Future  versions  will  build 

•  Systems  Engineering  Plan 

•  Measurement  and  Analysis  Plan 

•  Supplier  Agreement  Management  Plan 
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Training 


Systems  Center 
Charleston 


> Process  Improvement  and  CMMI® 

>  Systems/Software  Engineering  Classroom 


>Web  Based  Training  (WBTs) 
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Process  Improvement  Training 


Systems  Center 
Charleston 


•  Intro  to  Process  Improvement 

-  Over  800  people  trained 

-  Provided  via  WBT 

-  Now  Mandatory  for  all  employees 

•  CMMI® 

-  SEI’s  Intro  to  CMMI®  course  onsite 

-  SSC-C  Level  2  Processes 

-  875  people  trained 


• Project  Management/Project  Monitoring  &  Control 

-  625  people  trained 

•  Process-specific  Workshops  (CM,  QA,  REQ,  M&A) 

-  375  people  trained 


*  This  accounts  for  some  employees  attending  more  than  one  course 

Approved  for  release  to  the  public  - 15  October  2005 
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Systems  Engineering 
Fundamentals  Classes 


SPtWAR 


Systems  Center 
Charleston 

•  3-day  on-site,  classroom  course 

-  Based  on  SMU  SE  Masters  course 

-  Customized  to  incorporate  SSC-C  SE  process 

-  180  SSC-C  engineers  trained 

-  Classes  planned  every  2  months 


•  1-day  SE  for  Managers  course  added 

*  Intro  to  Software  Engineering  planned 


‘‘The  course  was  very  educational.  It  helped  me  relate  my  current 

fiti 


project  to  the  overall  system  it  was  a  part  of,  and  how  it  fits  in  with 
the  big  picture.  ” 


‘‘The  course  was  well  presented  and  accurately  covered  the  Systems 
Engineering  Design  Process  Fundamentals.  Continued/additional 
training  on  this  subject  is  critically  needed  for  this  command  to 
continue  to  develop  as  a  professional  engineering  organization.  ” 


Student  Feedback 
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PI  Web  Based  Training 


SBBMBR 


Systems  Center 
Charleston 


To  offer  Process 
Improvement 
training  to  more 
employees,  we 
developed  an  on¬ 
line  web  based 
tutorial  (PI-WBT) 
that  allows  students 
to  take  the  course  at 
their  own  pace  and 
to  receive  a 
certificate  and 
education  credit 
upon  course 
completion. 
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SE  101  Web  Based  Training 


Systems  Center 
Charleston 


•Introduction  to  Systems  Engineering 

- 1 0-module  web  based  training 

-Closely  aligned  to  SSC-C  SE  Process,  SE 
Fundamentals  Course,  ISO/IEC  15288  and  IEEE 
standards 

-  Includes  hotlinks  to  referenced  documentation 

•  Process  manuals,  policies,  standards 


ISO/IEC  15238 
Technical 
Processes 


CM  Ml1  se/sw 

Best  Practices 


SSC-C 

Standard  Processes 
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Systems  Center 
Charleston 

>  Accomplishments 

> Results  and  Measures 

>  Lessons  Learned 
>Going  Forward 
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What  We  Have  Accomplished 


Systems  Center 
Charleston 


•  Process  Focus 

-  Defined  Policies  and  Processes 

-  Aligned  with  DoD  and  SPAWAR  guidance 

-  Aligned  with  industry  standards  and  CMMI®  model 

-  Built  organization  structured  around  processes  and  process  improvement 

•  Training  is  Critical 

-  Providing  Fundamentals  of  Engineering  for  new  and  old  professionals 

-  Developed  web-based  training  for  “self-paced”  and  refresher  training 

-  Defining  a  structured  technical  career  development  path  for  engineers 

•  Tools  for  the  Engineers 

-  Developed  ePIan  Builder  application  to  generate  planning  documents 

-  Developed  templates,  checklists,  and  web-based  document  repositories 
to  link  standards  and  DoD  guidance  to  day-to-day  tasks  and  processes 


Early  and  persistent  Systems  and  Software  Engineering 
applied  to  programs  and  projects 
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Results  and  Measures 


Systems  Center 
Charleston 

•  Formal  process  improvement  policy  issued  in  2003 

-  Use  CMMI@to  evaluate  progress  against  best  practices 

•  Selected  pilot  projects 

-  Training  of  project  teams 

•  Informal  Appraisals,  Process  Reviews,  and  Document  Reviews 
to  measure  progress  and  identify  gaps 

-  Class  B/C  appraisals  of  selected  projects 

-  Define/review  project-specific  plans  and  procedures 

-  Ensure  the  processes  and  procedures  were  used 

•  Project-level  Formal  SCAMPISM  Appraisals  (Class  A) 

-  Evaluated  compliance  with  CMMI  ®  Maturity  Level  2  requirements 

-  8  projects  appraised  between  June  2004  and  February  2005 

•  Command-wide  appraisal  in  April,  2005 


SFmAR 
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Major  Milestone  -  Maturity  Level  2 


Systems  Center 
Charleston 


•The  first  SPAWAR  Systems  Center  to  achieve 
CMMI®  Maturity  Level  2  at  the  command  level 


SFMAR 


World  Class 
Systems 
Engineering 

Capability 

Maturity 

Model 

Integration 

(CMMI®) 


Systems 

Center 

Charleston 


MATURITY  LEVEL 

\  April  28,  2005 
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Systems  Center 
Charleston 


Lessons  Learned 


•  Senior  Management  support  is  critical  to  success 

•  Training 

-  Everyone  needs  to  be  engaged  -  “train  the  masses” 

-  Specific  training  for  process  owners/subject  matter  experts 

•  Utilize  Teams  (IPTs)  as  champions  of  specific  processes 

-  Multi-department  representation 

-  Change  agent  mentality 

-  Process  focused  charters 

•  Resource  Properly 

-  Implement  with  projects  that  want  to  improve,  can  benefit  from  efforts, 
and  that  recognize  own  weaknesses 

-  EPO  staff  provided  skilled  coaching,  resources,  support,  and  tools 

-  Project  members  learned  by  doing  and  maintaining 

•  Goals  and  Publicity 

-  Keep  goals  to  sizable  bites  (projects) 

-  Publicize  successes;  Share  best  practices 
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Summary 


Systems  Center 
Charleston 


V  Aggressive  SE  Program 

V  Industry  Standards 

-  Systems  Engineering  (SE) 

-  Software  Engineering  (SW) 

▼  Best 
Practices 


CMMI® 

ISO  9001 

Lean  Six 
Sigma 


SSC-C  SE  Revitalization 


Policy  /  Guidance  Training  /  Education  ■  Assessment  &  Support 


SPAWAR  SE 
Instruction  54xx.1 

SSC-C  SE  “ 

Process  Manual 


i 

Intro  to  PI  WBT 

CMMI®  Level  2 

* 

* 

SE  101  WBT 

CMMI®  Level  3 

- 1 - 

i 

SSC-C  SW 
Process  Manual 


ePIan  Builder 

- i - 

* 


SE  Fundamentals 

i  ~ 


□  Underway 
I  I  Implemented 


SW  Fundamentals 

i  ~ 


Certification  Program 


V  Training  - 1,300  people" 

V  Systems  Engineering 
Fundamentals  - 180 

Intro  to  SSC-C  PI 

-  CMMI®  Level  2 
Processes 

-  CMMI®  Level  3 
Processes 

-  SE/SW  Engineering 
Workshops 

-  Web-Based  Training 
(WBT)  for  Process  ' 
Improvement 


Integrated  Product 
Teams 

+  ~ 


SITC  -  Tools 


Lean  Six  Sigma 


Includes  industry 
partners 


V  Successes 

-  Command  Achieved 
CMMI®  Maturity 
Level  2  in  April  2005 

-  1st  SPAWAR  Systems  Center 
to  Achieve  CMMI®  Maturity 
Level  2 


Approved  for  release  to  the  public  -  1 5  October  2005 


Plans 

-  World  Class 
Systems  Engineering 

-  Support  Command 
Balanced  Scorecard 

-  April  2007  CMMI®  Maturity  Level  3 
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Going  Forward 


Systems  Center 
Charleston 

•  Develop  more  “how  to  ...”  guidance  and  tools 

-  ePIan  Builder,  an  interactive  web  application,  helps  build  required  plans. 

•  Currently  builds  PMP,  QA,  Configuration  Mgmt,  and  Requirements  Mgmt  plan 

•  Systems  Engineering  Plan,  Measurement  &  Analysis  Plan,  and  Supplier 
Agreement  Management  Plans  under  development 

-  Institutionalize  the  SE/SW  processes 

•  Emphasize  Formal  Reviews 

•  IPTs  -  expanding  beyond  CMMI®  &  Engineering  areas 

-  Expecting  more  integration  from  teams 

•  CMMI® 

-  SSC-Charleston  standard  process  with  Tailoring  Guidelines  for  all 
projects 

-  Projects  progressing  to  ML3 

-  Process  Improvement  tracked  at  department/project  level  using  self 
assessment  tool 

-  2  Balanced  Scorecard  measures  directly  related  to  CMMI® 


SFmAR 
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Thank  you  ! 


SFMAR 

▼ 

Systems  Center 
Charleston 


Any  Questions  ? 


Contact  Information: 

Michael  T.  Kutch,  Jr. 

SPAWAR  Systems  Center  Charleston 
Email:  michael.kutch@navv.mil 
Phone:  843-218-5706 - - 


Sandee  Guidry 

Technical  Software  Services,  Inc. 
Email:  sdauidrv@techsoft.com 
Phone:  850-469-0086 


Net-Centric 

Enterprise 
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What  Goes  On  Behind 
Closed  Doors 


Thomas  Lienhard  Jr 

Timothy  Davis 

Raytheon  Missile  Systems 
Systems  Engineering  Center 
1 151  E  Hermans  Rd,  MS  805/Cl 
Tucson,  AZ  85706 
520-794-81 55 

Thomas_G_Lienhard@raytheon.com 

tjdavis@raytheon.com 
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Perception  vs.  Reality? 


Raytheon 

Missile  Systems 


Perception  -  Appraisers  do  everything  in  their  power  to 
find  that  one  little  thing  that  ensures  that  the  organization 
will  fail  to  meet  it’s  goal 

Reality  -  Appraisers  really  do  WANT  the  organization  to 
achieve  their  CMMI  goal. 


It  is  up  to  the  organization  to  build  confidence  in  the 
appraisal  team  that  their  process  definition  and 
execution  meets  the  goals  of  the  Capability  Maturity 

Model  Integration  (CMMI) 
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So,  How  Is  This  Done? 


Raytheon 

Missile  Systems 


•  Assemble  the  “correct”  Appraisal  Team 

•  Be  prepared! 

•  Form  the  mindset  from  the  beginning 

•  Have  a  consistent  story 

•  Enable  the  appraisal  team  to  do  their  job 

•  Make  it  hard  for  the  appraisal  team  to  say 

•  Don’t  surprise  the  appraisal  team  during  the  SCAMPI 
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Assemble  the  “correct”  Appraisal  TealfftileSystems 

•  Choose  the  right  Appraisal  Team  Lead 

-  See  “Used  Cars  and  Lead  Appraisers”  (  by  Tim  and  Tom) 

•  Ensure  correct  Appraisal  Team  Knowledge  and  Skills 

-  Good  mix  of  discipline  and  functional  expertise 

-  Balance  Insider/Outsider  membership  on  the  team 

-  Monitor  team  dynamics 

•  Keep  the  Appraisal  Team  Together 

-  Focus  Process  Area  responsibilities 

-  Can’t  please  everybody  all  the  time,  so  please  the  appraisal  team 
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Be  Prepared! 
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Missile  Systems 


•  Be  committed  and  show  that  commitment 

•  Ensure  evidence  collection  is  complete,  concise  and 
adequately  organized 

-  Provide  detailed  mapping,  pointers,  and  comments 

•  Scrub  evidence  for  relevance  and  sufficiency 

-  Quantity  is  not  quality. 

-  In  fact,  quantity  usually  means  late  nights  and  VERY  grumpy 
appraisers!! 

•  Prepare  Participants 

-  Get  ready  for  the  face  to  face  interviews,  information  requests,  follow-up 
interviews,  preliminary  observations 


The  Appraisal  Team  should  be  verifying  not  discovering 
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Form  the  Mindset  From  the  Beginning 
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•  Demonstrate  an  understanding  of  the  model  and  appraisal 
methodology 

-  Appraisal  Planning  (scope,  model  representation,  and  schedule) 

•  Create  a  collaborate  environment 

-  Foster  a  win/win  relationship 

•  Schedule  and  Conduct  Intermediate  Reviews  with  Realistic 
Goals 

-  Class  Cs,  Class  Bs,  Internal  Readiness  Reviews,  SCAMPI 

-  Goals  must  be  consistent  with  the  type  of  review 

-  Don’t  forget  “Should-to-Shoulder”  reviews 

■  See  “Wasted  Days  and  Wasted  Nights”  (  by  Tim  and  Tom) 
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Have  a 


Story 
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Missile  Systems 


•  Paint  a  story  with  the  evidence: 

-  The  Appraisal  Team  is  not  clairvoyant  and  needs  the  story  presented 
clearly  through  the  evidence 

-  Weave  a  thread  through  and  across  the  process  areas 

-  Use  the  same  piece  of  evidence  for  multiple  practices 

•  Site  Brief 

-  TELL  how  your  processes  and  programs  satisfy  the  CMMI 

-  Get  credit  for  as  much  “affirmation”  as  possible 

•  Interviews 

-  TELL  how  your  processes  and  programs  satisfy  the  CMMI 

-  Appraisal  Team  needs  to  “hear”  what  was  seen 

Evidence  should  substantiate  the  site  brief 
Interviews  should  reinforce  the  evidence 
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Enable  the  Appraisal  Team  to  Do  Their  Job  M,ss,le Systems 

•  Have  dedicated  facilities  for  the  team 

•  Verify  tools  and  networking  are  working 
-  have  a  back-up  plan 

•  Appraisal  schedule  must  be  adhered  to  by  all  involved 

•  Have  evidence  collected,  inventoried,  and  VERIFIED! 

•  Do  not  burden  the  Appraisal  Team  with  40  pieces  of 
evidence  if  3  will  adequately  tell  the  story 

•  Have  sufficient  resources  to  respond  quickly  to  information 
requests  and  follow  up  interviews 
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Make  It  Hard  for  the  Team  to  Say 
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•  Stick  with  the  same,  or  a  core  set  of,  Appraisal  Team 
members 

•  Use  Appraisal  Team  as  consultants  up  to  the  SCAMPI 

•  Lead  the  Appraisal  Team  to  the  answer 

-  Map  the  evidence  to  the  practices  of  the  model,  have  detailed  pointers 

-  Use  clear  and  concise  comments  to  help  the  appraiser  understand  why 
the  evidence  is  relevant  to  that  practice 

•  Reinforce  evidence  with  the  Site  Brief  and  Interviews  -  tell 
the  same  story 

•  Have  a  Demo  if  it  helps  tell  the  story 
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Don’t  Surprise  the  Appraisal  Team  During 

■  i  .  .  1  ,  1  1  w  Missile  Systems 

the  SCAMPI 


•  Do  not  use  the  SCAMPI  to  validate  alternative  approaches 

—  Use  the  Intermediate  Reviews  to  gain  concurrence 


•  The  SCAMPI  should  be  little  more  than  a  rubber  stamp 

-  The  only  new  evidence  the  Appraisal  Team  should  see  is  to  address 
prior  weaknesses 


•  Remember  the  confidentiality  agreement 

—  Don’t  badger  the  Appraisal  Team  with  “How  Are  We  Doing?” 
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Questions 
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Team  of  Three 

How  to  get  Program, 

Functional,  and  Process 
Management  Working  Together 


Raytheon  Missile  Systems 
Mark  Marsh  and  Lety  Santillan 

November  16,  2005 
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520.794.2424 


520.794.2338 


Copyright  ©  2005  Raytheon  Company.  All  rights  reserved. 


10/24/2005  Page  2 


Organization  and  Accomplis h m e nts 
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Raytheon  Missile  Systems,  Headquarters  Tucson,  AZ 


Employees:  11,000 


’04  Sales:  $3.8  B 


World  Largest  Appraised  SEI 
CMMI  Level  3  Organization 
December  2004 


SW-CMM  Level  5  in 
November  2001 
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RMS  Approach  to  MA 
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•  Analyzed  the  PBA  findings  and  created  a  work  plan 

•  Benchmarked  against  other  CMMI  Level  3 
Raytheon  Organizations 

•  Conducted  peer  review  of  Organizational  MA  Plan 

•  Conducted  Several  focus  groups 

-  Program  Management 

-  Functional  Organizations 
—  CMMI  Experts 

-  Six  Sigma  Experts 

•  Concluded  Template  approach  best 

•  Team  of  Three  concept  is  born 

•  Measurement  Analysis  could  cover  more 
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Overview  of  M&A  Process 
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•  The  purpose  of  Measurement  and  Analysis  at  RMS:  The 

Measurement  and  Analysis  (M&A)  process  is  intended  to 
provide  information  to  the  projects  to  make  informed 
decisions  to  minimize  risk  and  ensure  project  success 
while  helping  to  support  the  overall  organization's  business 
objectives. 

•  The  purpose  of  Measurement  and  Analysis  in  CMMI:  The 

purpose  of  Measurement  and  Analysis  is  to  develop  and 
sustain  a  measurement  capability  that  is  used  to  support 
management  information  needs. 
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How  the  Right  Measures  Help  Teams 
Excel-  Harvard  Business  Review 
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•  The  overarching  purpose  of  a  measurement  system  should  be 
to  help  a  team,  rather  than  top  managers,  gauge  its  progress. 

•  A  truly  empowered  team  must  play  the  lead  role  in  designing 
its  own  measurement  system. 

•  Because  a  team  is  responsible  for  a  value-delivery  process 
that  cuts  across  several  functions  (like  product  development, 
order  fulfillment,  or  customer  service),  it  must  create 
measures  to  track  that  process. 

•  A  team  should  adopt  only  a 
handful  of  measures. 
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What  did  RMS  do? 
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•  Created  a  series  of  workshops  with  Program  Management 

-  Brought  to  the  table  the  critical  success  factors,  contractual 
requirements  and  lessons  learned  from  previous  programs 

-  Program  Management  Team  took  first  cut  at  what  goals, 
information  needs  and  measurements  objectives  for  the  program 
were. 

-Once  these  items  were  determined  then  metrics  were  selected 

-  Program  management  would  then  lead  a  workshop  with  IPT 
Leads  to  go  over  selections  get  buy-in  from  IPT  Leads 

-Once  the  program  management  and  IPT  Leads  agreed  on  the 
metrics  the  process  was  repeated  with  the  metric  collection 
points 

-Functional  and  process  representatives  were  selected  by 
program  team 
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Process  Improvement  Requires  Synergy 
between  Organization  and  Programs 
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•To  achieve  high  levels  of  process 
maturity,  the  organization  and 
programs  must  work  closely 
together 

-  New  process  at  the  organization 
level  need  to  be  deployed  to 
programs 

-  Best  practices  and  lessons 
learned  from  the  program  levels 
must  be  flowed  to  the  organization 
and  shared  across  programs 

-  Quantitative  management 
activities  need  infrastructure  to 
facilitate  metrics  collection  and 
analysis 
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Team  of  Three  (ToT)  Concept 
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•  Team  of  Four  and  Team  of 
X  concept  successful  at 
other  Raytheon  sites 

•  Adopted  the  concept  at  RMS 
in  2004 

•  Consistent  with  integrated 
product  team  approach 

•  Very  effective  mechanism 
for  process  improvement 


Vehicle  for  programs  to 
make  informed  decisions 
and  ensure  program 
success  while  helping  to 
support  the  overall 
organization's  business 
objectives 
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What  is  a  Team  of  Three  (ToT)? 
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•  A  team  that  supports  the  program  with 
process  deployment  and  analysis 

-  Team  goal  is  to  help  ensure  project 
success  while  helping  the  organization 
improve  over  time 

-  The  team  members  bring  a  broad 
perspective,  can  better  facilitate  sharing 
across  projects  and  help  the  organization 
improve  as  a  whole 

-  In  the  spirit  of  Integrated  Product  Teams 
(IPT) 

•  Also  the  primary  mechanism  for  process 
deployment  activities  on  projects 

-  Supports  the  organization’s  process 
improvement  efforts 

•  Should  not  impede  Program 


Copyright  ©  2005  Raytheon  Company.  All  rights  reserved. 


10/24/2005 


Page  10 


Team  of  Three  Members 
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•  Each  Team  of  Three  (ToT)  consists  of  (as  a  minimum): 

-  Program  Representative:  (“Chair”)  Program  Manager,  Chief 
Engineer,  or  a  Senior  Integrated  Product  Team  Lead 

-  Functional  Manager  (FM) 

□Typically  a  Department  Manager  (DM) 

□Systems,  “Integration  Test  &Analysis”  are  the  two  primary  functional 
organizations 

□  Radar,  Configuration  Management,  and  other  Centers  can  participate 
—  Process  Representative:  R6Sigma  Expert  or  IPDS@RMS  Expert 

•  Additional  representatives  may  attend  as  needed  (from  the 
program  or  other  organizations): 

-  Data  Management 

-  Quality 

-  Supply  Chain  Management 
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Working  as  a  Team  Contributes 
to  Program  Success 
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Team-of-Three 
Improved  Performance 


/ 


X 


Responsibilities: 

•  Chairperson 

•  Financial 

•  Technical 

•  Programmatics 

•  Process  planning  & 
implementation 

•  Metrics  collection 
&  analysis 


ENGINEERING  MGMT 

•  Functional  Management 
Responsibilities: 

•  Project  support 

•  Resource  provider 
(people,  tools,  training) 

•  Team  administration  & 
facilitation 

•  Process  improvement 

•  Cross-project  knowledge 

Copyright  ©  2005  Raytheon  Company. 


PROCESS 


•  Process  Representative 
Responsibilities: 

•  Project  support 

•  Process  expertise  & 
facilitation 

•  Data  Analysis  skills 

•  Lessons  learned  transfer 

•  Best  practices  transfer 

•  Process  improvement 

•  Cross-project  knowledge 
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Team-of-3  Individual  Responsibilities  for 
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The  Program  Representative 


Responsible  for  managing  the  activities  for  the  program 


•  Provides  the  current  project  status  and  metrics  to  the  ToT  for  analysis 

•  Implements  the  project’s  documented  process,  including  adjustments 
and  improvements 

•  Manages  the  project  plans  and  process  documents 

•  As  chair  of  the  ToT,  facilitates  team  meetings  and  works  to  improve  team 
effectiveness 

•  Ensures  meeting  minutes  are 
documented  and  submitted  to  the 

Process  Assets  Library  (PAL)  r  i 

•  Ensures  documents  are  under  r  1 

configuration  control 
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Team-of-3  Individual  Responsibilities  for 
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The  Functional  Manager 


Responsible  for  providing  adequate  resources 
to  the  program  in  a  timely  manner 


•  Provides  organizational  resources  to  the  project 

-  Examples:  trained  staff,  standard  project  tools 

-  Good  examples  of  “work  products” 

-  Not  just  their  “home  room”  but  advocate  to  all  other  functions 

•  Communicates  organization  goals  and  objectives 

•  Provides  insight  as  a  “higher  level  manager” 

has  purview  into  multiple  programs.  •  9  • 
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Team-of-3  Individual  Responsibilities  for 

The  Process  Engineer 
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\ 

Supports  process  deployment 

\ _ ) 


Provides  process  deployment  expertise  and  assistance 

Provides  the  organization  with  a  vehicle  for  sharing  lessons  learned  and 
process  changes  (when  necessary  and  appropriate) 

Facilitates  process-related  problem  resolution  and  process  improvement 

efforts 


Assists  with  evidence/artifact  collection  and  support  of  project 
evaluations 


Submits  project  data  (lessons  learned,  best  practices,  etc.)  to  the  PAL 
for  use  by  other  projects 

May  assist  program  with  preparation  &  review 
of  project  plans  and  process  documents 

-  Supplies  process  expertise 

-  Makes  process  improvement  recommendations 

-  Ensures  both  organizational  and  project  goals 
considered  in  Tailoring  Report 
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Team  of  Three 
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Project  Team  of  Three 


a 


Program  -Program  Manager  and/or  Chief  Engineer  or  delegate  (e.g.  System  IPT  lead) 
Functional  -  System  Engineering  and  IT  &  A  and  Configuration  Management  Dept  Mgrs/Section  Mgrs 

Process  -  R6Sigma  Expert  or  IPDS  expert 


.  System 
f  Level 


Functional 

Reviews 

Dept  Mgr 
SEPG 
SW  Lead 
Conf.  Mgmt 
SQE 


"\ 


,  Component 
Level 
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ToT  Adds  Value 
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•  Better  visibility  of  key  drivers 
(e.g.,  Productivity  and  other 
measures) 

•  Collaborative  risk  mitigation 

•  Timely  resolution  of  issues 
(more  proactive,  less  reactive) 

•  Eliminates  wasted  activities 
(no  “reinventing  the  wheel”) 


•  Institutionalized  processes 

•  In-Phase  containment  of 
defects 

•  Shared  Lessons  Learned  and 
Best  Practices 


Improved  Program 
Performance 


Improved 

Process/Product  Quality 


( 

Stronger  Tie  between  Programs  and  Functional  Organizations 

More  Successful  IPTs 
More  Predictable  Programs 


~\ 


y 
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ToT  Process  Improvement 
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•  Address  activities  identified  in  the 
program’s  process  improvement  plan 

•  Seek  process  improvements  for  areas 
identified  in  quantitative  management 

•  Sponsor  Process  Action  Teams  on  the 
program 

•  Identify  lessons  learned  and  best 
practices  to  share  with  rest  of 
organization 

•  Review  other  programs  lessons  learned 
and  best  practices  and  determine  if  they 
should  be  applied  to  this  program 

•  Prepare  for  appraisals,  customer 
reviews,  and  audits 
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Organization  Process  Teams 
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Project  Team 

Program  Management, 
Chief  Engineer, 

IPT  leads 


Team  of  Three  (ToT) 

Functional  Management, 
Program  Representative, 
Process  representative 


Define  &  deploy  the  project’s 
process 

Collect  &  analyze  metrics 
Implement  process 
improvements 
Manage  project  using  the 
metrics 

Develop  mitigation  plans  in 
response  to  unexpected  results 


Enterprise  Process  Grom 

EPG  rep 
Measurement  and  Analysis  Team 
Engineering  Process  Council 


Analyze  metrics  for  stability, 
trends  &  improvement 
opportunities  (org,  product  line  & 
project) 

Assign  resources  to  key  issues 
Assure  compliance  to  process 
standards 


Review  project  metrics  package 
Assure  key  issues  are  being 
addressed 

Provide  management  support  to 
projects 

Publish  IPR  (sum  of  project’s 
metrics  packages) 
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Benefits  of  Effective  Teams  of  Three 
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Improved  process 


-  Better  tailoring  and  deployment 

Improved  communication/collaboration 

More  consistency  across  projects 

Shared  lessons  learned  for  use  on 
other  programs 

Better  product  quality 

Improved  competitiveness 

Promotes  higher  maturity  processes 


“Working  as  a  Team  Fosters  Program  Success” 
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Why  do  all  this? 
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•  IPDS  @  RMS  provides  an  organizational  plan  and  template 
on  measures,  monitoring  and  control,  lessons  learned, 
review  of  program  and  process  data  with  a  functional 
representative 

•  CM  Ml  Level  3  looks  for  an  organization  process  that  has 
been  deployed  on  the  programs 

-  Measurement  Analysis  is  looking  for  a  disciplined  approach  to  metric 
selection  (“goal-question-metric”  philosophy) 

-  Program  Monitor  and  Control 

-  Generic  Practice  2.8  -  Process  Monitor  and  Control 

—  GP  2.10  -  Review  Status  with  Higher  Level  Mgmt 

—  GP  3.2  -  Collect  Improvement  Information 
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Using  the  Team  of  Three  for  CMMI 
evidence 
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•  For  Program  evidence  requirements,  the  Team  of  Three  proved  extremely 

useful. 

•  Each  program  was  required  to  provide  evidence  for  up  to  15  different 

process  areas. 

•  GP  2.8  Monitor  and  Control  the  Process 

-  Monitor  and  control  the  measurement  and  analysis  process  against  the  plan  for 
performing  the  process  and  take  appropriate  action. 

-  GP  2.8  was  satisfied  (metrics  package  and  minutes)  in  all  15  process  areas. 

•  GP  2.10  Review  Status  with  Higher  Level  Management 

-  Review  the  activities,  status,  and  results  of  the  measurement  and  analysis 
process  with  higher  level  management  and  resolve  issues. 

-  Satisfied  in  13  PA’s  from  Team  of  Three  evidence. 

•  GP  3.2  Collect  improvement  Information 

-  Collect  work  products,  measures,  measurement  results,  and  improvement 
information  derived  from  planning  and  performing  the  measurement  and 
analysis  process  to  support  the  future  use  and  improvement  of  the 
organization’s  processes  and  process  assets. 

-  Team  of  Three  was  a  major  contributor  for  GP  3.2  evidence  in  all  PA’s. 
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Questions? 

Thank  you  for  your  participation! 


Copyright  ©  2005  Raytheon  Company.  All  rights  reserved. 


10/24/2005 


Page  23 


Raytheon 

Network  Centric  Systems 


A  SCAMPI  Data  Review 
Improvement  Technique 

Getting  The  Big  Picture 

without  Pixelaflon 


Kent  McClurg 
Aaron  Clouse 


Agenda 


Raytheon 

Network  Centric  Systems 


•  Introduction 

•  Data  Prep  and  Data  Review  Issues 

•  An  Answer 

-  Process  Description 

-  Results 


Oct-05 


©Raytheon  2005 
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Introduction  - 1 


•  This  presentation  focuses  on  issues  associated  with  reviewing 
documents  during  a  SCAMPI  and  a  practical  solution  to 
resolving  these  issues. 

-  Issue  #1 :  The  context  of  the  applicable  information  is  often  not  clear 
since  the  appraisal  team  member  (ATM)  does  not  have  time  to  read  the 
entire  document  or  section  of  document  in  order  to  establish  the  context 

-  Issue  #2:  Jumping  from  one  document  to  another  or  from  one  section 
within  a  document  to  another  section,  in  an  attempt  to  determine  that  a 
practice  is  being  satisfied  is  not  very  efficient. 

•  We  will  explore  an  approach  that  you  may  find  improves  your 
appraisal  performance  significantly. 
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Issues  - 1 
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•  Data  does  not  always  address  practice  or  was 
difficult  to  understand 


-  Data  collector  understood  data, 
not  the  model 

-  Data  reviewer  understood  model, 
not  the  data 

•  Data  is  fragmented  -  too  many  pieces  needed 
to  demonstrate  compliance 


Practice 


Artifac 

:s  demonstrating  compl 

ance 

n 

rm 

n 
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Issues  -  2 
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•  A  single  artifact  often  addressed  multiple 
practices,  all  from  different  PAs  (and  mini¬ 
teams) 

Mini-teams  and  PAs 

Artifact(s) - J — J — J — J — J — J - ► 

•  Data  collectors  and  appraisers  have  different 
views  of  the  data 
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What  We  Had 
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Mini-Team 


Weak 

Association 


Strong 
Association^ 


Practice  Text 


Weak 
Association 


Direct  Artifact  #1 
Direct  Artifact  #2 
Direct  Artifact  #3 
Direct  Artifact  #4 

Indirect  Artifact  #1 
Indirect  Artifact  #2 
Indirect  Artifact  #3 
Indirect  Artifact  #4 


Data  Collectors 


Strong 

Association 


Disparate 

Artifacts 


Oct-05 


©Raytheon  2005 


6 


Issues  -  3 
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•  Not  possible  to  bring  appraisers  up  to  speed  on 
all  programs 

•  Not  practical  to  make  data  collectors  model 
experts 


SO  WHAT  CAN  WE 
DO? 
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An  Answer  1 
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•We  considered  our  options  with  two  concepts 
at  hand 

-  What  we  COULD  NOT  do,  and 

-  What  we  COULD  do 
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An  Answer  2 
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•  What  we  could  not  do: 

-  We  could  not  make  all  of  the  data  collectors 
experts  in  understanding  and  interpreting  the 
CMMI 

•  Too  expensive 

•  Not  value  added  for  the  organization 

-  We  could  not  make  program  experts  out  of  the 
appraisal  team  members 

•  Not  available  for  in  depth  orientation  on  all  programs 
included  in  the  appraisal 

•  No  value  to  the  programs 
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An  Answer  3 
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•  What  we  could  do: 

-  Establish  a  method  (process  description  and  tools) 
that 

•  Enables  programs  to  adequately  and  accurately  address 
a  practice 

•  Provide  appraisal  team  members  with  a  clear  view  of 
the  data  in  the  context  of  the  program(s) 

•  Allow  the  appraisal  team  to  more  efficiently  and 
effectively  review  the  data 
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Process  Description  Assumptions 

•  Only  major  artifacts  would  be  addressed  by  this  process 

-  Program  plan 

-  SEMP 

-  Quality  Plan 

-  Configuration  Management  Plan 

-  etc. 

•  In  general,  artifacts  that  have  Specific  Practices  that  need  to  be 
addressed  AND  Generic  Practices  that  need  to  be  addressed 

-  Ex:  PPQA  SPs  and  GP2.9 

-  Ex:  CM  SPs  and  GP2.6 

•  Each  artifact  is  mapped  to  all  of  the  applicable  PAs/Practices 
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Process  Description 

•  Focus  is  on  reviewing  the  data  by  artifact  rather  than 
by  PA 

-  Enabler  (e.g.,  checklist)  must  be  established  such  that  it 
can  be  sorted  by  artifact  element  OR  by  PA/Pr 

•  Enabler  is  sorted  by  artifact  and  appropriate  sections 
of  the  enabler  (checklist)  are  provided  to  the  selected 
reviewers 

•  Appraisers  review  the  artifact  using 
the  checklist 
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Enabler  Relationship  to  Artifact 


Configuration 

Management 

Plan 
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The  Enabler  (aka  “The  Checklist”) 

•  Checklist  Content 

-  Artifact  or  plan  name 

-  Artifact  Component 

•  The  plan  content  is  derived  from  the  organizations  standard 
plan/artifact  template 

-  Maturity  Level  and  Process  Area  (e.g.,  2-PP) 

-  Practice  identifier  (e.g.,  SP2.1) 

-  Place  Plan/Section  Identifier 

-  Place  to  capture  the  need  to  ask  a  question 

-  Place  for  reviewer  comments 
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Checklist  For  Data  Review 
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Artifact  or 
Plan  Name 

Artifact  Component 

Level  -  PA 

Practice 

Plan/ 

Section 

Ask  ?? 

Reviewer 

Comments 

Project  Data/ 
Config  Mgmt 
Plan 

Configuration  of  Product  Requirements 
Baseline  Identified 

2-REQM 

GP2.6 

Pgm  xyz 
Program 

CM  Plan, 
section 

3.4.1 

How 
does  xyz 
? 

This  section 
identifies  the 
right  thing. 

Project  Data/ 
Config  Mgmt 
Plan 

Configuration  of  Product  Requirements 
Traceability  Identified 

2-REQM 

GP2.6 

Project  Data / 
Config  Mgmt 
Plan 

Configuration  of  Other  Project  Planning 
Artifacts  is  Identified 

2-PP 

GP2.6 

Project  Data / 
Config  Mgmt 
Plan 

Identification  of  data  items  to  be 
included  in  the  project’s  data 
management  plan. 

2-PP 

SP2.3 

Project  Data/ 
Config  Mgmt 
Plan 

Data  management  strategy 

2-PP 

SP2.3 

Project  Data / 
Config  Mgmt 
Plan 

Data  formatting  standards 

2-PP 

SP2.3 

Project  Data/ 
Config  Mgmt 
Plan 

Data  collection  objectives 

2-PP 

SP2.3 
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Raytheon 

Network  Centric  Systems 

Using  “The  Checklist”  i 


•  Review  the  artifact  against  the  checklist  indicating  compliance 
as  appropriate  (write  the  location  of  observed  evidence  in  the 
Plan/Section  col.) 

•  Make  appropriate  comments  if  necessary 

-  Handy  during  the  discussions  with  the  team 

-  Also  provides  responsible  team  members  with  additional  information 

•  Indicate  if  there  are  any  questions  that  you  would  like  to  ask 
during  the  interview  sessions 

•  Enter  the  data  into  the  master  Checklist 

•  Sort  the  checklist  by  PA/practice 

•  Provide  the  information  to  the  appropriate  mini-team 
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Where  We  Ended  Up 


Raytheon 

Network  Centric  Systems 


Strong 

Association 


Mini-Team. 

Mini-Team, 


Mini-Team 


Practice  Text 
For  Multiple 


PAs  and  Practices 


Weak 
Association 


Weak 

Association 


Direct  Artifact  #1  Ref#1 
Direct  Artifact  #1  Ref#2 
Direct  Artifact  #1  Ref#3 
Direct  Artifact  #1  Ref#4 

Indirect  Artifact  #1  Ref#1 
Indirect  Artifact  #1  Ref#2 
Indirect  Artifact  #1  Ref#3 
Indirect  Artifact  #1  Ref#4 


Data  Collectors 


Strong 

Association 


Homogeneous 
Artifact  - 
Multiple 
Practices 
Addressed 
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Conclusion 


Raytheon 

Network  Centric  Systems 


•  Mini-teams  have  vetted  data  from  all  of  the  major 
plans/artifacts 

•  The  appraisal  TEAM  has  a  better  “picture”  of  each  plan 

•  Mini-team  members  have  insight  into  evidence  beyond  their 
own  assigned  areas 

•  Broader  knowledge  of  “in  context”  information  for  making 
judgments  concerning  strengths,  weaknesses,  etc. 

•  References  to  artifacts  can  be  checked  against  the  checklist  for 
acceptability,  rather  than  having  to  review  the  same  artifact 
multiple  times 
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Raytheon 

Network  Centric  Systems 
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The  ROI  Dashboard© 
Understanding  the  Benefits  of  CMMI 


Tom  McGibbon,  CSDP 

Data  &  Analysis  Center  for  Software,  Director 

775  Daedalian  Dr. 

Rome,  NY  13441 
315.334.4933 
tom.mcgibbon@itt.com 
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ROI  Dashboard© 


http://www.thedacs.com/databases/roi/ 


rw-c  The  Data  &  Analysis  Center  for  Software 


/  A 


ROI  Dashboard' 


Submit  n  Study 


In  response  to  increasing  interest  and  attention  from  the  software  engineering  and  software  acquisition 
community  for  benefits  data  from  software  technical  and  management  improvements,,  the  DACS  pressnts  the 
ROI  Dashboard©,  The  ROI  Dashboard©  augments  and  updates  the  DACS  Report  "A  Businas?  t  ese  foi  3oFtwara 
process  improvement*  with  the  latest  published  data  on  benefits,  The  roi  Dashboard©  graphically  displays 
open  and  publicly  available  data  and  provides  standard  statistical  analysis  of  the  data  To  learn  more  about  the 
features  and  usage  of  the  ROI  Dashboard©  please  read  the  overview. 


Step  1: 

Select  the  improvement  areas  you  ere  interested  in 
examining  (select  up  to  four  by  using  the  control  key).  Note; 
Improvements  are  split  into  two  groups  i  those  with  extensive 
benefit  data  and  those  with  only  limited  data,  “fo  viaw  what 
improvements  organizations  have  implemented  concurrently, 
please  view  our  impmVBfngm  matrix  TO  view  more 
details  about  cmm  and  cmmi  Improvements  click  mere, 


—  Extensive  Data  Available - 

Agile  Developmeni 

CMM  Software  Process  Improvement 

CMMI  Process  Improvement 

Clean  room 

inspections 

Measurement  Program 

PSP/TSP 

Reuse 

—  Limited  Data  Aval  I  able - 


Step  2 : 

Whet  type  of  display  are  you  interested 

in? 

0  Bom  Plot  fd stalls') 

C  Bar  Plot  f details) 

O  TeMt  fdetails) 


Submit 


If  you  have  data  about  the  benefits  from  software  process  improvements  at  your  organization  and  would  lika  to 
submit  them  for  Inclusion  In  the  roi  Dashboard©,  Please  submit  a  case  study  fit  vou  have  concerns  regarding 
privacy  or  proprietary  information,  please  read  about  our  data  collection  ooliiv).  If  you  submit  data,  vou  are  entitled 
to  receive  a  free  gift:  either  our  "A  Business  Case  for  Software  Process  Improvement*  resort  or  the  0 ACS  DOD/TT 
Acronym  List  on  CD  ROM. 
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Objective:  Transition  from 
Anecdotal  Evidence  to 
Industry  Trends 

Captures  10  Years  of 
Open  and  Public  ROI 
Data  from  Industry  and 
Acquisition  Organizations 

Organizes  and  Displays 
Data  from  Similar 
Improvements  and 
Benefits 
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Published  Data 
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ROI  Dashboard8 


Submit  e  Case  Study 


Results  for  CMM  Software  Process  Improvement. 

On  any  graph  below,  "mouse-over"  a  displayed  line  segment  to  see  details  of  individual  observations,  "Click"  on  a  line  segment  of 
interest  to  see  further  details  about  the  individual  observations.  "Click"  inside  the  box  to  display  a  timeline  of  the  improvements.  The 
numbers  above  each  graph  represent  the  number  of  data  points  for  that  graph. 


Back  to  search  Contribute  your  data  through  our  online  survey  or  via  email. 

ROI 


CMM 

Software 

Frocess; 

Improvement 


Reduction  in  Rework 

100-1 


SO- 


crs 

CD 

O 

OS 

T3 


60- 


o 


TMM 


Impact  on  Productivity 


Attribute  Displays: 

•  ROI 

•  Reduction  in  Rework 

•  Impact  on  Productivity 

•  Impact  on  Quality 

•  %  Defect  Reduction 

•  %  Defects  Found 

•  Impact  on  Schedule 

•  Cycle  Time 

•  Schedule  Variance 

•  Reduction  in  Project  Cost 

•  Cost  of  Improvement 
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Box  Plots 


Impact  on  Quality  (%  defect  reduction) 


Software 

Frocess 

Improvement 


'1  DACS  RQI  Dashboard  -  Microsoft  Internet  Explorer 


□  ® 


Box  Hot 


li 


. 


15 


Extreme  vilue  -  eieater  than  3  box  knEths  above  th?  hin?e 
approxinatin?  the  third  quart  ik 


Gutlkr-i  -  ?reater  than  1 3  box  len?thi.  above  the  hin?e 
approxinatin?  the  third  quart  ik 

L ar  ?e  it  Ibserved  value  wi thin  1 5  lex  len Ethi  trail  hinEe 
V-hiilzer 

Data  point {  ii  within  1. 5  lex  len?thi  from  hinge 
Hm?e  appro  ximat  in?  t  hird  quart  it 
Bex 
Mean 
Median 

Hinge  approximating  riTEt  quart ik 
^hidzer 

Smalleit  observed  \alue  within  1.5  box  kngthi  trail  hinge 


Outlkr  -  greater  than  1  5  box  lengths  below  the  hinge 
approxinatin?  the  tirit  quart  ik 


Extreme  value  -  greater  than  3  box  kngthi.  below  the  hinge 
approxinatin?  the  tirit  quart  ik 


For  further  explanation  of  boxplots  and  hinges  please  see  (Tukey  1977)  1  W. 
Tukey,  Exploratory  Data  Analysis,  Addison  Wesley. 
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ROI  Dashboard©  Analyzes  Benefit 


Data  from  Best  Practices 


rw~c  The  Data  &  Analysis  Center  for  Software 


Bar  Plot 


3  http:AAvww.thedacs.CQm/datab. . .  f^J 


Impact  on  Quality  (%  defect  reduction) 


Software 

Frocess 

Improvement 


Bar  Plot 


-Larzis-t  ct5-=n."5c  valw 

ET  § 


Mean 


ET  = 

It 

S' 


-iJiia'll-Est  oksarvad  valus 


*  =  [>L  -  ar,  —  .  .  -  xm  };k 


.  I  ~  Or*)3  -  tiH-3c  f 
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Sample  ROI  Dashboard©  Bar  Plot 
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Text 

rw~c  The  Data  &  Analysis  Center  for  Software 


ROI  Dashboard 


Improvement:  CMM  Software  Process  Improvement 


Overview 


Submit  a  Case  Study 


Metric 

Total  Data 
Points 

Minimum 

Maximum 

Median 

Mean 

Standard 

Deviation 

25th 

Percentile 

75  th 

Percentile 

Impact  on  Quality  (% 

1 

90 

%  defects 
found 

100  %  defects 
found 

94  %  defects 
found 

94.67  %  defects 
found 

5.03  %  defects 
found 

90  %  defects 
found 

100  %  defects 
found 

of  defects  found) 

ROI 

18 

0.14 

Ratio 

19  Ratio 

6  Ratio 

5.9  Ratio 

3.99  Ratio 

4.3  Ratio 

6.8  Ratio 

Impact  on  Quality  f% 

24 

-6 

%  defect 
reduction 

100  %  defect 
reduction 

50  %  defect 
reduction 

47.9  %  defect 
reduction 

33.16  %  defect 
reduction 

22  %  defect 
reduction 

72  %  defect 
reduction 

defect  reduction) 

Impact  on  Cycle  Time 

12 

-19 

%  decrease 

90  %  decrease 

43  %  decrease 

40.5  %  decrease 

36.09  % 
decrease 

14.5  % 
decrease 

70  %  decrease 

Impact  on  Schedule 

10 

-50 

%  decrease 

98  %  decrease 

46  %  decrease 

43.3  %  decrease 

40.73  % 
decrease 

33  %  decrease 

67  %  decrease 

Variance 

Impact  on 

27 

-5 

% 

improvement 

350  % 

improvement 

37  % 

improvement 

81.78  % 
improvement 

98.06  % 
improvement 

13  % 

improvement 

100  % 

improvement 

Productivity 

Reduction  in  Rework 

6 

28 

%  decrease 

73  %  decrease 

36  %  decrease 

41.5  %  decrease 

16.71  % 
decrease 

30  %  decrease 

46  %  decrease 

Reduction  in  Proiect 

2 

18 

%  decrease 

20  %  decrease 

19  %  decrease 

19  %  decrease 

1.41  %  decrease 

13  %  decrease 

20  %  decrease 

Cost 

Cost  of  the 

2 

2 

%  of  total 
effort 

3.15  %  of  total 
effort 

2.58  %  of  total 
effort 

2.58  %  of  total 
effort 

0.81  %  of  total 
effort 

2  %  of  total 
effort 

3.15  %  of  total 
effort 

Improvement 
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ROI  Dashboard© 

into 

25 -i 


Provides  Visibility 
Data 

ROI 


20- 
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CMM 

Software 

Process 

Improvemei 


Reduction  in  Rework 


60- 


40- 


CMM 

Software 

Frocess 

Improvement 


Reduction  in  Project  Cost 


20.5- 
20- 
1  9  5- 


75th  Percentile:  6.27 

In  a  1994  SEI  ROI  study,  GTE 
Government  Systems 
Corporation  observed  a  6.8  to 
1  ROI  from  the  overall  SPI 
effort.. 

In  moving  from  CMM  Level  2  to 
Level  5,  Motorola  invested 
$90,180  and  saved  $611,200 
on  rework,  for  a  ROI  of  578%. 

Based  on  a  sample  in  1990  of 
six  large  real-time  embedded 
projects,  Raytheon's  Software 
Systems  Laboratory  achieved 
a  6.7  annual  ROI  while  moving 
from  CMM  Level  1  to  CMM 
Level  3,  Investment  was 
approximately  $1  million  per 
year,  and  the  sampled  projects 
used  about  58%  of  SSL  labor 
resources.  Approximately  $9.2 
million  in  reduced  rework  was 
saved  by  1990,  of  which  $4.48 
million  were  saved  in  1990, 
($15.8  million  was  saved  by 
1992.) 

Oklahoma  City-ALC  gathered 
ROI  information  for  18  of  44 
software  process 
improvements.  They  invested 
$462,100  for  a  return  of 
$2 .935M,  a  ROI  of  6.35  to  1. 


Tipact  on  Productivity 


ment 


tst  of  the  Improvement 
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Details  Available  When  Needed 


i-vy~c  The  Data  &  Analysis  Center  for  Software 


ROI  Dashboard4 


Submit  a  Case  Study 


Fact  Information 


From  1992  to  1999,  Thomson  -  CSF  implemented  the  following  Improvements: 
-  CMM  Software  Process  Improvement 


BACKGROUND: 


■  Company  currently  known  as  Thales 

■  European  worldwide  leader,  $7  billion  in  yearly  revenues 

■  50,000  employees,  50%  engineers,  30%  outside  France 


OBSERVED  RESULTS: 

Thomson  -  CSF  observed  the  following  changes: 


ROI  Dashboard© 


■  The  overall  cost  (investment)  of  the  improvement  was  -2,8  to  -3.5  %  of  Software  Effort  measured  after  the 
improvements(CMM  Level  2  Organizations) 

■  The  overall  cost  (investment)  of  the  improvement  was  -2  %  of  Software  Effort  measured  after  the 
improvements(CMM  Level  3  Organizations) 

■  The  overall  cost  (investment)  of  the  improvement  was  31,000  hours  measured  after  the  improvements(CMM 
Level  1  to  CMM  Level  3) 

■  The  cost  of  training  was  2  days  /  year  measured  after  the  improvementsfTraining  required  to  reach  CMM  Level 
2) 

■  The  cost  of  training  was  3  days  /  year  measured  after  the  improvementsfTraining  required  to  reach  CMM  Level 
3) 

■  Estimated  savings  in  development  cost  was  -57,000  to  -100,000  hours  measured  after  the  improvements(CMM 
Level  1  to  CMM  Level  3,  1992-1997) 

■  The  Return-on-Investment  (Cost  Savings/Cost  of  Improvement)  was  4.3  Ratio  measured  after  the  improvements 
(CMM  Level  2,  1993-1995) 

■  The  Return-on-Investment  (Cost  Savings/Cost  of  Improvement)  was  3.75  Ratio  measured  after  the 
improvements(CMM  Level  3,  1996-1997) 

■  The  deviation  in  actual  cost  compared  with  estimated  cost  was  -130  to  -150  %  measured  before  the 
improvements  and  0  %  measured  after  the  improvements(CMM  Level  1  to  CMM  Level  3) 

■  The  number  of  defects  delivered  to  the  customer  was  -130  to  -150  %  measured  before  the  improvements  and 
0  %  measured  after  the  improvements(CMM  Level  1  to  CMM  Level  3) 

■  Total  defects  observed  was  -50  %  measured  after  the  improvements(In  Peer  Code  Reviews  (80  %  of  code)) 

■  The  number  of  defects  delivered  to  the  customer  was  44  measured  before  the  improvements  and  9  measured 
after  the  improvements(1993-1996,  from  2  CMM  Level  1  projects  to  4  CMM  Level  2  projects) 

■  The  number  of  defects  delivered  to  the  customer  was  9  measured  before  the  improvements  and  3.4  measured 
after  the  improvements(1996-1997,  from  4  CMM  Level  2  projects  to  18  CMM  Level  3  Projects) 

■  Increase  in  Schedule  Performance  Index,  where  SPI  is  the  ratio  at  the  scheduled  completion  date  of  the  amount 
of  work  performed  and  all  scheduled  work,  was  50  %  measured  after  the  improvementsfCMM  Level  1  to  CMM 
Level  2) 
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CMMI  Inspections 

PSP 

Reuse 

CMMI  Inspections 

Process 

Process 

/ 

Improvement 

Improvement 

TSP 

PSP 

/ 

TSP 


Reuse 
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%  defect  reduction 


Combined  -  ROI  Dashboard© 


Impact  on  Quality  (%  defect  reduction) 


Impact  on  Cycle  Time 


CMMI  Inspections  FSP  Reuse 

Process  / 

Improvement  TSP 


CMMI  Reuse 

Process 

Improvement 
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Improvement  Area  Matrix 


rw~c  The  Data  &  Analysis  Center  Jot  Software 


ROI  Dushboaid* 
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rw^c 

The  Data  &  Analysis  Center  for  Software 

✓  X* 

DAT  1"\  _ 1© 

IVVJi  uasuuuaiu  Overview  FAQ 

Submit 

a  Case 

Study 

Improvement  Area  Matrix 

The  following  table  shows  which  pairs  of  improvements  are  commonly  performed  together  by  organizations  currently  in 
the  DACS  ROI  Database,  Each  cell  contains  the  total  count  of  records  found  in  our  database  (where  the  improvement 
pair  is  defined  by  the  row  and  column).  You  can  view  the  matching  records  by  clicking  on  the  total  count, 

Agile 

Development 

CMM 

Software 

Process 

Improvement 

CMMI 

Process 

Improvement 

Cleanroom 

ISO 

9001 

Inspections 

Measurement 

Program 

PSP  / 
TSP 

Reuse 

Agile 

Development 

10 

0 

0 

0 

0 

0 

0 

0 

0 

CMM 

Software 

Process 

Improvement 

0 

63 

1 

1 

1 

1 

0 

| 

I 

CMMI 

Process 

Improvement 

0 

9 

23 

0 

0 

0 

0 

2 

0 

Cleanroom 

0 

% 

0 

5 

0 

0 

0 

0 

I 

ISO  9001 

0 

i 

0 

0 

1 

0 

0 

0 

0 

Inspections 

0 

7 

0 

0 

0 

19 

0 

0 

0 

Measurement 

Program 

0 

0 

0 

0 

0 

o. 

3 

0 

0 

PSP  /TSP 

0 

2 

2 

0 

0 

O’ 

0 

11 

0 

Reuse 

0 

1 

0 

1 

jr 

O' 

0 

0 
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Access  Detailed  CMM/CMMI  Data 
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Submit  a  Case  Study 


In  response  to  increasing  interest  and  attention  from  the  software  engineering  and  software  acquisition 
community  for  benefits  data  from  software  technical  and  management  improvements,  the  DACS  presents  the 
ROI  Dashboard©.  The  ROI  Dashboard©  augments  and  updates  the  DACS  Report  "A  Business  Case  for  Software 
Process  Improvement"  with  the  latest  published  data  on  benefits.  The  ROI  Dashboard©  graphically  displays 
□  pen  arid  publicly  available  data  and  provides  standard  statistical  analysis  of  the  data.  To  learn  more  about  the 
features  and  usage  of  the  ROI  Dashboard©  please  read  the  overview. 


Step  1: 

Select  the  improvement  areas  you  are  interested  in 
examining.  In  the  list  below  ROI  data  is  classified  by  the 
organizations  final  maturity  level  while  an  asterick  represents 
any  previous  CMM  or  CMMI  level.  Note:  Improvements  are  split 
into  two  groups:  those  with  extensive  benefit  data  and  those 
with  only  limited  data.  You  are  currently  viewing  CMM/CMMI 
related  data  only,  to  view  all  data  please  click  here. 


Step  2: 

What  type  of  display  are  you  interested 
in? 

<;>  Box  Plot  (details) 

O  Bar  Plot  (details) 

O  Text  (details) 


Extensive  Data  Available 


Achieving  CMM  L2 
Achieving  CMM  L3 
Achieving  CMM  LA 
Achieving  CMM  LB 
Achieving  CMMI  L£ 

Achieving  CMMI  L3 
Achieving  CMMI  LB 
-  Limited  Data  Available 


Achieving  CMMI  LA 


Submit 


If  you  have  data  about  the  benefits  from  software  process  improvements  at  your  organization  and  would  like  to 
submit  them  for  inclusion  in  the  ROI  Dashboard©,  please  Submit  a  Case  Study  (if  you  have  concerns  regarding 
privacy  or  proprietary  information,  please  read  about  our  data  collection  policy).  If  you  submit  data,  you  are  entitled 
to  receive  a  free  gift:  either  our  "A  Business  Case  for  Software  Process  Improvement"  report  or  the  DACS  DOD/IT 
Acronym  List  on  CD  ROM. 
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Analysis  of  ROI  Dashboard©  Data 


As  of  1 0/6/05 

Agile  Development 

CMM  SPI 

CMMI  PI 

Cleanroom 

Inspections 

Meaurement  Program 

PSP/TSP 

Reuse 

ISO  9001 

Total 

Number  of  Reports 

10 

63 

23 

5 

19 

3 

11 

19 

1 

154 

Quality:  %  Defect  Reduction 

4 

24 

16 

5 

1 

5 

1 

1 

57 

Quality:  %  Defects  Found 

3 

1 

1 

6 

2 

13 

Quality:  Reduction  in  Rework 

6 

1 

1 

8 

Total  Quality  Related 

4 

33 

18 

1 

12 

1 

7 

1 

1 

78 

HHHHI 

Cost:  Productivity  Impacts 

3 

27 

9 

2 

2 

1 

10 

54 

Cost:  Reduction  in  Program  Costs 

2 

2 

1 

i 

1 

7 

Total  Cost  Related 

3 

29 

11 

2 

3 

i 

2 

10 

61 

nnnm 

Schedule:  Impact  on  Cycle  Time 

2 

12 

4 

i 

10 

29 

Schedule:  Schedule  Variance  Impact 

10 

1 

2 

13 

Total  Schedule  Related 

2 

22 

5 

i 

2 

10 

42 

^ m 

nnnm 

nmmi 

ROI:  Return  on  investment 

18 

5 

i 

14 

2 

2 

3 

1 

46 

nnnm 

lillllil 

Cost  of  Improvement 

2 

1 

i 

1 

5 

■HH 

■til 

Total  Benefits  Observed 

9 

104 

39 

5 

30 

5 

13 

25 

2 

232 

16 
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ROI  Dashboard© 
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ryr'c  The  Data  &  Analysis  Center  for  Software 


ROI  Dashboard 


Overview 


Submit  a  Case  Study 


Improvement:  CMMI  Process  Improvement 


Metric 

Total  Data 
Points 

Minimum 

Maximum 

Median 

Mean 

Standard 

Deviation 

25th 

Percentile 

75th 

Percentile 

ROI 

5 

3 

Ratio 

13.3  Ratio 

5  Ratio 

6.46  Ratio 

3.93  Ratio 

4  Ratio 

9.65  Ratio 

Impact  on  Cycle 

4 

15 

%  decrease 

50  %  decrease 

29  %  decrease 

30.75  % 
decrease 

16.19  % 
decrease 

17.5  %  decrease 

44  %  decrease 

Time 

Reduction  in  Rework 

1 

60 

%  decrease 

60  %  decrease 

60  %  decrease 

60  %  decrease 

0  %  decrease 

0  %  decrease 

0  %  decrease 

Impact  on  Quality 

16 

0.5 

%  defect 
reduction 

95  %  defect 
reduction 

48.5  %  defect 
reduction 

46,97  %  defect 
reduction 

29.52  %  defect 
reduction 

25.5  %  defect 
reduction 

67  %  defect 
reduction 

(•%  defect  reduction) 

Impact  on 

9 

5 

% 

improvement 

73  % 

improvement 

30  % 

improvement 

34.33  % 
improvement 

25.9  % 
improvement 

9  % 

improvement 

59  % 

improvement 

Productivity 

Impact  on  Schedule 

1 

50 

%  decrease 

50  %  decrease 

50  %  decrease 

50  %  decrease 

0  %  decrease 

0  %  decrease 

0  %  decrease 

Variance 

Impact  on  Quality 

1 

93 

%  defects 
found 

93  %  defects 
found 

93  %  defects 
found 

93  %  defects 
found 

0  %  defects 
found 

0  %  defects 
found 

0  %  defects 
found 

f%  of  defects  found) 

Reduction  in  Project 

2 

20 

%  decrease 

40  %  decrease 

30  %  decrease 

30  %  decrease 

14.14  % 
decrease 

20  %  decrease 

40  %  decrease 

Cost 
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Detailed  Summary  Data 


1 


16- 

14- 

12- 

10- 

o 

^  8- 
LL 

6- 

4- 

2- 

n 

1 

■ 

ROI 

2 

2 

1 - 1 

U 

1 

i 

i  ■  i 

Achieving 

Achieving 

Achieving 

CMMI 

CMMI 

CMMI 

L2 

L3 

L5 

1 00  “| 


80- 

-i— ■ 

£= 

CD 

I  60- 
> 

O 

g-  40  H 
20- 


0 


Impact  on  Productivity 


Achieving 

CMMI 

L3 


Achieving 

CMMI 

L4 


Achieving 

CMMI 

L5 
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Detailed  Summary  Data 


2 


Impact  on  Quality  (%  defect  reduction) 


o 

o 


T3 

di 


O 

£ 

di 

T3 


120- 

100- 

80- 

60- 
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Achieving  Achieving 
CMMI  CMMI 

L2  L3 


1 

- 1 - 1 - 

Achieving  Achieving 
CMMI  CMMI 

L4  L5 


50-i 
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Oj 

cti  30- 
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<U 
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Impact  on  Cycle  Time 


Achieving 

CMMI 

L2 


Achieving 

CMMI 

L3 


Achieving 

CMMI 

L5 
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CMM  vs.  CMMI 


ROI 


Impact  on  Productivity 


CMM  CMMI 

Software  Frocess 

Process  Improvement 

Improvement 


Impact  on  Quality  (%  defect  reduction) 


CMM  CMMI 

Software  Frocess 

Process  Improvement 

Improvement 


CMM  CMMI 

Software  Frocess 

Frocess  Improvement 

Improvement 


Impact  on  Cycle  Time 


CMM  CMMI 

Software  Frocess 

Frocess  Improvement 

Improvement 
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Challenges  in  Open  Reported  Data 


•  Data  reported  from  commercial  organizations  being 
reported  is  inherently  competition  sensitive 

•  Only  successes/improvements  reported.  Few  failures. 

•  Some  observations  are  vague 

•  Some  authors  only  report  notional  data 

•  Data  not  adequately  defined/quantitative,  e.g.  “Near 
Zero  Defects  Delivered.” 

•  Benefits  reported,  but  not  cost  of  the  improvement 

•  Some  only  report  averages.  How  to  combine  with 
specific  case  studies? 

•  Variability  in  units  and  definitions 

•  Inconsistent  use  of  terms  in  reporting.  What  part  of  the 
lifecycle  was  measured? 


ROI  Dashboard©  21 

2005  CMMI  Tech  Conf  ©  2005  by  ITT  Industries,  AES;  Tom  McGibbon 


rw~c 

✓  vy 


Building  the  Business  Case 


•  CMMI  has  demonstrated  with  quantifiable 
evidence  of  improvements  in  cost, 
schedule,  and  quality 

•  Use  data  in  process  modeling 

•  Compare  your  data  to  Dashboard  data 
-  Does  it  agree?  If  not,  why  not? 

•  Build  simple  spreadsheets  for  what  if 
analysis 
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Next  Steps 


•  Need  More  Data:  CMMI  and  other 

•  Need  Feedback  from  You  on  the  ROI 
Dashboard©  for  Problems  & 
Enhancements 

•  Coordination  with  SEI  on  CMMI  Data 
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Thank  You! 


ROI  Dashboard© 

2005  CMMI  Tech  Conf 


Tom  McGibbon,  CSDP 

Data  &  Analysis  Center  for  Software,  Director 

775  Daedalian  Dr. 

Rome,  NY  13441 
315.334.4933 
tom.mcgibbon@itt.com 
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Rapidly  Achieving 
Measurable  ROI 
Using  Early  Defect 

Detection 

NDIA  2005  CMMI  Technology  Conference 

November  16,  2005 

Timothy  G.  Olson,  President 

Quality  Improvement  Consultants,  Inc.  (QIC) 

(760)  804-1405  (Office) 

Tim.Olson@qic-inc.com 

WWW.qiC-inc.COm  ©  1994-2005  by  Process  Assets,  LLC  (PAL) 
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Presentation  Objectives 

“Raise  the  standard”:  describe  best-in-class  early 
defect  detection  and  measurable  results. 


Provide  motivation  for  performing  early  defect 
detection. 

Describe  early  defect  detection  principles,  and 
describe  a  best-in-class  early  defect  detection 
process. 

Describe  how  to  estimate  and  measure  ROI  using 
defect  dollarization. 

Answer  any  questions. 
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Agenda 

Why  use  Early  Defect  Detection? 
World-Class  Early  Defect  Detection 
What  are  In-Process  Inspections? 
Defect  Dollarization  and  ROI 
Summary 

Questions  and  Answers 
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Industry  Standard 
Cost  Ratio  to  Fix  a  Defect 


Defects  cost  less  to  fix  when  detected  earlier  in  the  process 
$  80-1 00X 


1 - i - 1 - i - 1 - ►time 

Requirements  Design  Implementation  Test  Release 
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EDD  Shortens  the  Schedule 
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Requirements  Design  Implementation  Test  Release 


SCHEDULE 


•  Adapted  from  Fagan,  M.  “Advances  in  Software  Inspections”,  IEEE  Transactions  on  Software  Engineering,  July  1986 
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Why  use  Early  Defect  Detection? 
World-Class  Early  Defect  Detection 
What  are  In-Process  Inspections? 
Defect  Dollarization  and  ROI 
Summary 

Questions  and  Answers 
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World-Class  Strategies 


DEFECT 

PREVENTION 


NUMBER 

OF 

DEFECTS 


EARLY 


Req.’s  Design  Code  umi  Test  Release 

Test 


•  Slide  adapted  from  Olson,  “A  Software  Quality  Strategy  for  Demonstrating  Early  ROI”,  SSQ  Journal,  May  1995. 


Used  with  Permission  and  Licensed  by  Quality  Improvement  Consultants,  Inc.  (QIC) 


Page  7 


World-Class  Quality _ 

*  Best-In-Class  EDD  Benchmarks 


MEASUREMENT 

WORLD-CLASS  BENCHMARK 

Costs  of  Poor  Quality 
(COPQ) 

Reduced  from  33%  to  under  10% 

(Goal:  Cut  COPQ  in  half  in  5  years) 

Defect  Removal  Efficiency 

70-90%  defect  removal  before  test 

Post-Release  Defect  Rate 

Six  Sigma  (3.4  defects  per  million) 
Software  Benchmark: 

0.01  Defects  per  KSLOC) 

Productivity 

Doubled  (e.g.,  in  5  years  at  -20%  a  year) 

Return  on  Investment 

5:1  -15:1  ROI 

Schedule  /  Cycle  Time 

Reduced  by  10-25%  (e.g.,  per  year) 
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EEVVA 

Review  Purpose/Type 

Education 

Communication;  Raise  Issues  (e.g.,  Walkthroughs) 

Evaluation 

Raise  issues;  Consensus  (e.g.,  Peer  Reviews) 

Verification 

Verify  req.s;  Remove  defects  (e.g.,  Inspections) 

Validation 

Meet  user  needs  (e.g.,  User  Groups) 

Assurance 

Product  and  process  assurance  (e.g.,  Audits) 

•Adapted  from  Ebenau,  Software  Inspection  Process,  McGraw  Hill,  1994 
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Example  EDD  Strategy: 
Defect  Removal  Efficiency  (DRE) 


NUMBER 

OF 

DEFECTS 


Inspect 

100% 

SyRS 


Inspect 

100% 

SRS 


Peer 

Review 

Designs 


Inspect 

100% 

High 

Risk 


Peer  | 

Review  |Early  I  CM:  Problem 
Other  liestinqj  Reports 


Requirements  Design  Implementation  Testing  Release 


•  Slide  adapted  from  Olson,  “A  Software  Quality  Strategy  for  Demonstrating  Early  ROI”,  SSQ  Journal,  May  1995. 
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^  EDD  Strategies 

Use  the  EEVVA  Model  to  ensure  that  all  reviews 
have  a  clear  objective. 


Use  multiple  EDD  processes  to  achieve  earlv  defect 
detection  and  track  defects  to  closure  (e.g.,  CM, 
Peer  Reviews,  Inspections,  Walkthroughs,  Audits, 
Early  Testing,  etc.) 


Requirements  are  critical  documents.  Formally 
inspect  all  requirement  documents  to  remove  as 
many  defects  as  possible. 


Formally  inspect  all  high  risk  designs  and  code  to 
remove  as  many  defects  as  possible. 


Other  documents  (e.g.,  design,  code)  may  be  peer 
reviewed  and/or  sampled. 
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Why  use  Early  Defect  Detection? 
World-Class  Early  Defect  Detection 
What  are  In-Process  Inspections? 
Defect  Dollarization  and  ROI 
Summary 

Questions  and  Answers 
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What  are  In-Process  Inspections? 

The  purpose  of  in-process  inspections  is  to  detect 
defects  early  in  the  process  in  order  to  reduce 
rework  and  costs,  and  to  increase  quality  and 
productivity. 

In-Process  Inspection: 

A  formal  process  for  verifying  intellectual  products 
(in-process)  by  manually  examining  a  work  product, 
a  piece  at  a  time,  by  small  teams  of  trained  peers  to 
detect  defects,  to  ensure  that  the  product  is  correct 
and  conforms  to  standards,  product  specifications, 
and  requirements. 

•Adapted  from  Ebenau,  Software  Inspection  Process,  McGraw  Hill,  1994 
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Characteristics 

Inspections 

Reviews 

Walk-throughs 

Goal 

Identify  defects 

Reach  consensus 
Raise  issues 

Reach  consensus 
Raise  issues 

State  of  Work 
Product 

Final  draft 

Work  in 
progress 

Work  in 
progress 

Process/ 

Measurements 

Formal/ 

Required 

Informal/ 

None  required 

Informal/ 

None  required 

Checklists/ 
Error  Detection 

Required/ 

Defects 

classified 

Not  required/ 

Not  required 

Not  required/ 

Not  required 

Participants 

Moderator;Reader; 
Recorder;  Author; 
Inspectors 

Author; 

Reviewers 

Author; 

Reviewers 

Process 

Owner 

Moderator; 

Independent 

verification 

Author 

Author 
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Inspection  Process  Model 
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Some  world-class  EDD  characteristics  are: 

•  Well-Defined  process  (e.g.,  roles) 

•  Process  models  (e.g.,  Role/Flow,  ETVX,  SADT) 

•  Well-Defined  measurements 

-  Internal  metrics  (e.g.,  preparation  rate) 

-  External  metrics  (e.g.,  productivity) 

•  Data  driven  checklists 

•  Tailored  to  the  organization  and  to  the  projects 

•  Data  analysis,  statistics,  and  reliability 

•  Interfaces  to  other  processes  (some  examples): 

-  Configuration  Management 

-  Defect  Prevention 


•  Reference:  World-Class  Inspection  Process  Guide,  Olson,  Timothy  G.,  1994 
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Why  use  Early  Defect  Detection? 
World-Class  Early  Defect  Detection 
What  are  In-Process  Inspections? 
Defect  Dollarization  and  ROI 
Summary 

Questions  and  Answers 
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'w  EDD  ROI  Assumptions 

According  to  industry  data,  in-process  inspections 
average  about  7:1  ROI. 

Historically,  industry  tests  in  quality  (e.g.,  80%  of 
all  defects  are  found  in  test). 

According  to  industry  data, .defects  cost  10-20 
times  more  when  found  in  test. 

Once  a  defect  is  identified,  testing  processes  can 
take  5-20  hours  to  fix  and  verify  per  defect. 

Once  a  defect  is  identified,  in-process  inspections 
take  about  0.5-1  hours  to  fix  and  verify  per  defect. 
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^  ROI  Goal/Questions/Metrics 

Goal:  Measure  ROI  (both  estimated  and  actual) 

Key  Questions: 

1.  How  much  does  a  defect  cost  in  each  phase  of 
the  process  (e.g.,  design  vs.  test  vs.  release)? 

2.  What  is  the  defect  removal  rate  of  the  verification 
processes  for  each  phase  (e.g.,  inspections,  peer 
reviews,  testing)? 

3.  For  each  project: 

•  how  many  total  defects  (estimated  and  actual)? 

•  how  many  total  defects  in  each  phase  of  the 
process  (estimated  and  actual)? 
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w'  Key  ROI  Metrics 

Key  ROI  metrics  to  compare  verification  processes: 

•  Total  percentage  of  project  (effort  or  cost) 

•  Work  product  size  by  phase  (e.g.,  total  pages, 
KSLOC,  etc.) 

•  Number  of  defects  (total  and  by  phase) 

•  Defect  density  (e.g.,  defects  per  page  or  KSLOC) 

•  Effort  (person  hours)  per  page  or  KSLOC 

•  Effort  per  defect  (fully  loaded  processes) 

•  Effort  per  defect  (after  defect  is  identified) 

•  Defect  removal  efficiency  (DRE) 

•  ROI  =  Cost  Reduction/Investment  (annually) 
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'w  Simple  ROI  Example 

Calculate  ROI  using  defect  dollarization  for  100 
similar  small  projects  (100  projects  per  year). 

Defect  ratio  is  10X  (0.5  hours  to  fix  defect  early  in 
the  process  and  5  hours  to  fix  a  defect  in  test). 
NOTE:  10X  is  usually  the  best  case.  Many  times  it 
is  15X  or  20X. 

Our  simple  example  will  assume  no  previous  EDD, 
and  75%  defect  removal  efficiency  after  installing 
EDD. 

Our  example  will  assume  $100,000  was  spent  on 
EDD  training. 
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Example  Pre-EDD  (0%  DRE): 
Defect  Dollarization 


Estimated  100  projects  annually  * 
[100  defects  *  5  hours]  *  $100  = 

$5,000,000 

NUMBER 

OF 

DEFECTS 


I 


I 


I 


100%  of 
Measured 
Defects 
Found  in 
Testing 


Req.’s  Design 


Code  Unit  Test  Release 
Test 


•  Slide  adapted  from  Olson,  “A  Software  Quality  Strategy  for  Demonstrating  Early  ROI”,  SSQ  Journal,  May  1995. 
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Example  Post-EDD  (75%  DRE): 
Defect  Dollarization 


NUMBER 

OF 

DEFECTS 


100  projects  annually  * 
[75  defects  *  .5  hours] 
*$100=  $375,000 


100  programs  annually  * 
[25  defects*  5  hours]  * 
$100=  $1,250,000 


Req.’s  Design 


Code  Unit  Test  Release 
Test 


•  Slide  adapted  from  Olson,  “A  Software  Quality  Strategy  for  Demonstrating  Early  ROI”,  SSQ  Journal,  May  1995. 


Used  with  Permission  and  Licensed  by  Quality  Improvement  Consultants,  Inc.  (QIC) 


Page  23 


World-Class  Quality _ 

^  Average  7:1  ROI 

Pre-EDD  MOO  Projects): 

•  100  defects  *  5  hours  =  500  hours  (50,000  hours) 

•  500  hours  *  $100  =  $50,000  ($5,000,000) 

Post-EDD  (100  Projects): 

•  75  defects  *  0.5  hours  =  37.5  hours  (3,750  hours) 

•  37.5  hours  *  $100  =  $3,750  ($375,000) 

•  25  defects  *  5  hours  =  125  hours  (12,500  hours) 

•  125  hours  *  $100  =  $12,500  ($1,250,000) 

Investment:  $475,000  ($100K  Training  +  EDD  Process) 

Return:  $5,000,000  -  $1 ,725,000  =  $3,275,000 

ROI:  $3,275,000  /  $475,000  =  7:1  (100  Projects  Annually) 
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Why  use  Early  Defect  Detection? 
World-Class  Early  Defect  Detection 
What  are  In-Process  Inspections? 
Defect  Dollarization  and  ROI 
Summary 

Questions  and  Answers 
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Summary 

Most  of  industry  “tests  in  quality”,  and  testing  in 
quality  is  expensive. 

Early  defect  detection  saves  money,  improves 
productivity,  and  reduces  cycle  time. 

Best-in-class  results  are  defect  removal  efficiency 
of  70-90%  before  testing. 

In-Process  inspections  average  7:1  ROI  which  can 
be  realized  in  less  than  6  months. 

Early  defect  detection  and  defect  prevention  are 
world-class  strategies. 
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Overview 


•  The  remise  of  this  session 

•  The  Problems 

•  The  Result 

•  A  Suggested  Solution  and  Practice 
Implementation  Indicator  Descriptions  (PIIDs) 

•Recommendations 

•Typical  results  in  performing  an  organization’s  first 
appraisal  (Backup  Slides) 
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The  Premise 


The  majority  of  early  CMMI®  presentations  at  previous 
conferences  have  been  by  organizations  already  familiar  with  the 
CMM  and  CMMI 

-  Many  already  at  CMM  of  CMMI  Maturity  Level  3  or  higher 

-  Many  are  Transition  Partners  with  the  SEI  and  have  internal  experts, 
appraisers,  and  instructors  that  are  familiar  with  PHDS 

However  other  companies  that  do  not  have  this  experience 
reported  on  their  experience  transitioning  to  the  CMMI  and 
preparing  for  their  first  appraisal.  This  presentation  reviewed 
four  small  and  one  large  organization  attempting  to  implement 
the  CMMI.  Several  high  level  hurdles  to  overcome. 

■  Ability  and  effort  to  understand  the  new  model 

■  Relative  effort  to  convert  their  processes  to  be  fully  CMMI  compliant 
using  the  PIIDs 

-  Effort  to  prepare  for  and  undergo  a  SCAMPIsm  appraisal  (A,  B  or  C) 
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The  Problem 
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Some  Basic  Problems  in  Implementing  CMMI 


Companies  (Large  and  Small)  reviewed  are  starting  from  the 
following  premise: 


■  They’ve  have  not  paid  the  price  to  wrap  up  the  learning  curve  on  the 
model 


-  They  have  no  infrastructure 

-  They  have  no  lessons  learned 

■  No  monies  allocated  to  sufficiently  institutionalize  practices  and 
processes 

■  Lack  of  sufficient  resources 


■  Unreal  time  frame  expectations  from  senior  management  to  achieve  a 
Maturity  or  Capability  Level 


Lack  of  senior  management  involvement  to  ensure  progress  is  being 
made 
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Some  Basic  Problems  in  Implementing  CMMI 
(Cont’d  -1) 


■  Project  Managers  are  part  of  the  technical  project 

■  Process  training  (project  management,  process  improvement  and 
other  appropriate  training  is  not  delivered  in  a  timely  fashion.  This 
includes  for  example: 

-EPG  training 
-MSG  training 
-Process  training 
-Estimation  techniques 
-Quality  Assurance  auditing  techniques 

-  Measurement  training 

-  Project  management  training 
-Building  schedules 
-Intro  to  CMMI  course 
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Some  Basic  Problems  in  Implementing  CMMI 
(Cont’d  -2) 

-  EPG  members  do  not  understand  their  role;  e.g.,  interfacing  to  the 
projects  and  the  MSG 

■  People  with  the  wrong  skill  sets  are  members  of  the  EPG. 


-No  software/systems  engineering  background 

-No  interpersonal  skills 

-No  writing  skills 

-No  team  building  skills 

-Should  have  at  least  taken  the  Intermediate  course  and  passed  it. 
Of  the  five  organizations  in  this  study  five  members  of  the  EPG 
took  the  Intermediate  Concepts  of  CMMI  course  from  the  SEI  and 
only  two  individuals  on  different  EPG’s  passed  it. 

■  They  do  not  know  what  it  takes  to  succeed  in  an  appraisal  and 
prepare  their  PHD  documentation  and  personnel  carefully  for 
participation  in  the  SCAMPI  process 
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The  End  Result 
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The  End  Result 


If  you  are  just  starting  out  in  process  improvement,  you  will 
likely  have  quantitatively  different  experiences  than  you  expect 

"  Effort  to  understand  the  CMMI  model  will  be  greater  without  the  CMM 
experience  as  a  basis.  This  is  particularly  true  with  organizations 
dealing  with  services  (Help  Desks,  Training  Service  projects) 

-  Mapping  your  practices  to  the  model  and  preparing  the  PUDS  will  be 
incomplete  because  your  early  understanding  of  the  model  is 
incomplete 

■  Preparing  for  an  appraisal  will  consume  a  lot  of  resources(  e.g.  in  one 
organization  preparing  and  successfully  passing  a  SCAMPI  Class  A 
appraisal  took  over  2,000  hours  including  project  personnel  from  3 
projects) 
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The  End  Result  (Cont’d  -1) 


■  Notifying  the  Lead  Appraiser  eight  months  ahead  of  time  but  having 
no  time  for  the  Lead  Appraiser  to  review  the  CMMI  Process 
Improvement  Program.  As  a  result  when  the  Lead  Appraiser  does 
walk  through  the  door  for  a  SCAMPI  Class  C  he/she  finds  that  the 
entire  process  improvement  program  was  placed  in  the  hands  of  two 
EPG  representatives.  The  results  of  the  SCAMPI  Class  C  showed  a 
number  of  process  problems  relating  to  the  Specific  Practices  in  the 
Level  2  areas: 


No  processes/practices  to  deal  with  RTM’s,  scoping,  estimating, 
measuring,  quality  assurance  audits  etc.  Then  the  question  is 
asked  “  Do  you  think  we  can  meet  our  management  goal  of 
attaining  a  Maturity  Level/Capability  Level  in  30  days? 
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The  End  Result  (Cont’d  -2) 


■  Asking  a  Lead  Appraiser  to  perform  a  SCAMPI  Class  A  in  30  days? 

-  What  about  those  Practice  Implementation  Indicator  Descriptions? 

-  Dealing  with  the  results  of  the  first  appraisal  can  be  catastrophic  if 
you  aren’t  prepared 
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Problems  Preparing  for  the  Appraisal  -  PIIDs 
(Cont’d  -1 ) 


CMMI  appraisals  are  verification-based  appraisals  which  rely  on 
the  organization  identifying  Process  Implementation  Indicator 
Descriptions  (PIIDs)  and  collecting  artifacts  to  prove  the 
practices  are  implemented-  not  a  discovery  based  effort  (CBA-IPI 
or  a  SCE) 

Immature  organizations  have  trouble  understanding  how  to  fill  out  the 
PIIDs 

■  Direct  and  Indirect  Artifacts  are  confusing 

-Object  oriented  practices  vs.  action  oriented  practices 

-One  practice’s  direct  artifact  can  be  another  practice’s  indirect  artifact 

-  A  shotgun  approach  is  used  (list  as  many  artifacts  as  you  can  think 
of) 

-  Usually  none  of  the  artifact(s)  address  the  practice 

-  Too  general  in  terminology,  not  specific  enough  to  show  the  practice 
is  really  implemented 

-Meeting  minutes,  emails,  action  items 

•  Actual  examples  of  the  above  don’t  cover  the  practice  in  question 
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Problems  Preparing  for  the  Appraisal  -  PIIDs 
(Cont’d  -2) 


■  Appraisal  teams  will  spend  an  inordinate  amount  of  time 
separating  the  appropriate  artifacts  from  the  rest  and 
“discovering”  the  real  state  of  process  implementation  with 
immature  organization 
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Problems  Preparing  for  the  Appraisal/PIIDS 
Politics 


Organizations  just  starting  out  have  a  culture  change  problem  to 
deal  with 

"  Managers  don’t  want  to  air  their  dirty  linen  so  they  are  reluctant  to  be 
appraised 

■  Project  teams  haven’t  really  bought  into  the  CMMI  yet  so  they  are 
reluctant  to  get  involved 

-  Everybody  is  busy  and  the  organization  doesn’t  want  to  disturb  the 
projects 

-  PIIDs  will  be  filled  out  by  a  third  party  who’s  trained  in  the  CMMI  and  in 
filling  out  PIIDs  but  without  specific  project  knowledge,  resulting  in 
wasted  effort  trying  to  deal  with  the  project  team  to  get  information 

or 

-  PIIDs  will  be  filled  out  by  project  team  members  that  may  not  have  been 
fully  trained  on  what  is  needed,  resulting  in  repeated  requests  for 
more/better  information  on  the  PIIDs  by  the  appraisal  team 

Organizations,  where  we  have  performed  appraisals,  tend  to  fill  out  the 
PIIDs  for  PP  and  PMC  with  “Project  Plan”  listed  as  the  Direct  Artifact  for 
each  practice.  The  DAR  Practice  filled  out  the  PIIDS  with  the  DAR  Plan 

for  each  practice. 
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A  Suggested  Solution  and  PIIDs 
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A  Suggested  Solution 


A  typical  first  step  in  most  process  improvement  efforts  (after 
the  Introduction  to  CMMI  training)  is  to  map  your  practices 
against  the  CMMI  to  determine  your  objective  evidence  (PHD) 
gaps 

"  Most  organizations  haven’t  documented  their  current  practices 

-  It  can  be  hard  to  tell  what  you  do 

-  It’s  easy  to  assume  just  because  you  know  of  a  process,  it’s  widely 
used  in  the  organization 

-  Most  Level  1  organizations  that  have  documented  processes  don’t 
follow  them 

-They  are  not  widely  communicated 

-They  are  not  integrated  into  a  whole  project  “system”  with  supporting 
training,  templates,  and  management  encouragement 

-They  are  abandoned  at  the  first  sign  of  trouble 
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Using  the  Practice  Implementation  Indicator 
Descriptions  (PIIDs)  to  help  the  team 


Practice  Implementation  Indicator 
Descriptions  (PIIDs) 

Practice  implementation  indicators  are  “footprints” 
which  are  evidence  of  the  conduct  or 
implementation  of  a  practice. 

SCAMPI  appraisals  use  practice  implementation 
indicators  as  the  focus  to  verify  practice 
implementation. 

Verifying  practice  implementation  is  the  review  of 
Objective  Evidence  to  determine  whether  a 
practice  is  implemented  within  a  project  and/or 
organization. 

THE  PROBLEM  :  “We  really  didn’t 
understand  the  dynamics  of  the  CMMI 
model  until  we  tried  to  prepare  the 
PIIDS  for  the  appraisal.” 

CarneoeMellcm 

©2005CSM  ndia  Conference  Software  Engineering  Institute_ a 


CSM 


THE 

CENTER  FOR 

SYSTEMS  MANAGEMENT 

INTEGRATING  PROCESS  &  PERFORMANCE 


Practice  Implementation  Indicator  Description 
Types 


Direct  Artifacts 
Indirect  Artifacts 
Affirmations 

PIIDs  include  documents  as  well  as  information  gathered  from  interviews  with 
managers  and  practitioners. 

Indicators  provide  a  useful  and  reliable  way  of  predicting  that  something  is 
present  or  true. 

Example:  Automobile  fuel  gauge 
Pros: 

can  highly  simplify  repetitive  and  costly  operations 
can  be  great  time  savers 

Cons: 

can  be  misleading 
can  be  wrong 
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Direct  Artifacts 


•  Tangible  output(s)  resulting  directly  from  implementation  of  a  specific 
or  generic  practice 

•  Integral  part  of  verifying  practice  implementation 

•  May  be  explicitly  stated  or  implied  by  the  practice  statement  or 
associated  informative  material 

Examples: 

■  Typical  work  products  listed  in  CMMI  practices 

-  Target  products  of  an  “establish  and  maintain”  specific  practice 

-  Documents,  deliverable  products,  training  materials,  etc. 
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Indirect  Artifacts 


•  Artifacts  that  are  a  consequence  of  performing  a  specific  or  generic 
practice  or  that  substantiate  its  implementation,  but  which  are  not  the 
purpose  for  which  the  practice  is  performed. 

•  That  is,  an  artifact  exists  but  there  is  no  indication  of  where  it  came 
from,  who  worked  to  develop  it,  or  how  it  is  used. 

•  Examples: 

Meeting  minutes,  review  results,  status  reports,  performance  measures 
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Affirmations 


•  Oral  (interviews)  or  written  statements  confirming  or  supporting 
implementation  of  a  specific  or  generic  practice 

•  Usually  provided  by  the  practice  implementers  or  other  stakeholders 

•  May  include  interviews  that  are  face-to-face,  video  conference  or 
teleconference,  or  equivalent 
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Recommendations 
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Recommendations  for  preparing  PIIDs  from  performing 
SCAMPI’s  with  mature/immature  organizations 


■  Mature  organizations  usually  have  employed  one  of  the 
following  techniques  to  resolve  PHD  issues: 

-  Provide  a  “Process  Improvement”  Coach  to  help  the  project  teams 
resolve  the  direct  vs.  indirect  objective  evidence  problem 

-  An  Organization  Process  Group  that  provides  direction  through  a 
series  of  workshops 

-“Tiger  Team”  approach  with  projects 

-  Bimonthly  reviews  of  PIIDS  by  PPQA 

-  Organizational  PPQA  provides  oversight  in  conjunction  with 
reviews  with  Project  Status  reviews  to  senior  management 


monthly 
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Recommendations  for  preparing  PUDS  from  performing 
SCAMPI’s  with  immature/mature  organizations 


-  Most  Immature  organizations  do  not  dedicate  the  resources  to 
Process  Improvement  to  effectively  fill  out  the  PHDS 

-  Work  allocation  is  in  the  range  of  10-20%  of  time  allocated  to  process 
improvement-  the  rest  is  billable/project  work 

-  No  understanding  of  the  model 

-  The  EPG  has  little  experience  in  process  improvement 

-  Do  not  have  the  CMM  background  to  fall  back  on  in  terms  of  what  is 
involved  in  process  improvement 

-Time  allowed  by  management  to  fill  out  PUDS  is  minimal 
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Typical  First  Appraisal  (Class  C  w/5  projects) 


Specific  Practices  by  Process  Area 
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Typical  First  Appraisal  (Class  B  w/5  projects) 


P 
G  r 
e  a 
n  c 
e  t 
r  i 
i  c 
c  e 
s 


GP  2. 1  Establish  an  Organizational  Policy 

GP  2.2  Plan  the  Process 

GP  2.3  Provide  Resources 

GP  2.4  Assign  Responsibility 

GP  2.5  Train  People 

GP  2.6  Manage  Configurations 

GP  2.7  Identify  and  Involve  Relevant  Stakeholders 

GP  2.8  Monitor  and  Control  the  Process 

GP  2.9  Objectively  Evaluate  Adherence 

GP  2. 1 0  Review  Status  with  higher  Level  Management 


Process  Areas 
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Different  Views  -  Specific  Practices  SCAMPI  Class  B  - 
Can  you  tell  which  organizations  were  prepared? 


PP  PMC  SAM  REQM  VER  CM  PPQA  MA 


PP  PMC  SAM  REQM  VER  CM  PPQA  MA 


PP  PMC  SAM  REQM  VER  CM  PPQA  MA 


SP  2.1 
SP  2.2 
SP  2.3 
SP  2.4 
SP  2.5 
SP  2.6 
SP  2.7 


SP  2.1 
SP  2.2 
SP  2.3 
SP  2.4 
SP  2.5 
SP  2.6 
SP  2.7 


q  i 

■ 

SP  3.1 
SP  3.2 
SP  3.3 


SP  3.1 
SP  3.2 
SP  3.3 


SP  3.1 
SP  3.2 
SP  3.3 


SP  1.6 
SP  1.7 


SM 


SP  2.1 
SP  2.2 
SP  2.3 
SP  2.4 
SP  2.5 
SP  2.6 
SP  2.7 
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Any  Questions  or  Comments? 


Backup  -Sample  PIIDs  Results 
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Interpreting  the  Results 


Fully 

Implemented  (FI) 

•Direct  artifacts  present  and  appropriate 
•Supported  by  indirect  artifact  and/or  affirmation 
•No  weaknesses  noted 

Largely 

Implemented  (LI) 

•Direct  artifacts  present  and  appropriate 
•Supported  by  indirect  artifact  and/or  affirmation 
•One  or  more  substantial  weaknesses  noted 

Partially 

Implemented  (PI) 

•Direct  artifacts  absent  or  judged  inadequate 
•Artifacts  or  affirmations  indicate  some  aspects  of  the 
practice  are  implemented 
•One  or  more  substantial  weaknesses  noted 
*Projects  that  have  not  reached  the  point  in  the  life  cycle  to 
have  produced  the  necessary  direct  artifacts  are  rated  PI  and 
this  would  be  accounted  for  when  the  instantiations  are 
aggregated  at  the  OU  level  practice  rating. 

Any  situation  not  covered  by  above 
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Different  Views  -  Generic  Practices 


GP  2.1 

Establish  an  Organizational  Policy 

GP  2.2 

Plan  the  Process 

GP  2.3 

Provide  Resources 

GP  2.4 

Assign  Responsibility 

GP  2.5 

Train  People 

GP  2.6 

Manage  Configurations 

GP  2.7 

Identify  and  Involve  Relevant  Stakeholders 

GP  2.8 

Monitor  and  Control  the  Process 

GP  2.9 

Objectively  Evaluate  Adherence 

GP  2.10 

Review  Status  with  Higher  Level  Management 

GP  2.1 

Establish  an  Organizational  Policy 

GP  2.2 

Plan  the  Process 

GP  2.3 

Provide  Resources 

GP  2.4 

Assign  Responsibility 

GP  2.5 

Train  People 

GP  2.6 

Manage  Configurations 

GP  2.7 

Identify  and  Involve  Relevant  Stakeholders 

GP  2.8 

Monitor  and  Control  the  Process 

GP  2.9 

Objectively  Evaluate  Adherence 

GP  2.10 

Review  Status  with  Higher  Level  Management 
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INTEGRATING  PROCESS  &  PERFORMANCE 


SCAMPI  Class  A  Level  3  SE/SW  Goal  Profile  (3rd  effort 
towards  achieving  a  Maturity  Level  3) 


CSM 


ORGANIZATIONAL  UNIT  CHARACTERIZATION  AND  GOAL/PA  RATING  MAP 

(actual  Assignments) 
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Evaluating  the  Impact  of 
New  Tools  and  Technologies 
Using  Simulation 


David  M.  Raffo,  Ph.D.,  Portland  State  University 
Tim  Menzies,  Ph.D.,  Portland  State  University 


Portland  State 

UNIVERSITY 

Agenda 

•  Motivation 

•  Learned  Defect  Detectors  Highlights 

•  Process  Simulation  Highlights 

•  Model  Overview 

•  Three  Scenarios  and  Results 

•  Conclusions 
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Motivation 

•  Good  new  technologies  are  wasted 

-  unless  there  is  a  compelling  business  case  to  use  them 

•  Without  such  a  case: 

-  Managers  not  convinced 

-  No  reallocation  of  scarce  resources 

•  Good  technology:  data  mining  defect  detectors 

-  increased  PDs  (probability  of  detection) 

-  Lower  PFs  (probability  of  false  alarm) 

-  Lower  inspection  effort  (more  time  for  other,  more  specialized 
methods 

•  This  talk: 

-  The  business  case 

-  Developed  via  process  simulation 


Portland  State 
University 


Data  mining  defect  detectors 


4 


•  Data  miners  learn  detect  detectors  from 
static  code  measures  (  McCabe  and 
Halstead)at  the  module  level. 

-  Not  perfect:  widely  deprecated  (Shepherd, 
Fenton,  and  others) 

-  Adequate  as  partial  indicators  (but  watch  that 
false  alarm  rate) 


has  defect 

No 

Yes 

detector  silent 

A 

B 

C 

D 

detector  triggered 

accuracy= 

pd  =  i 

pf 

prec  = 

Effort  = 

(a+d)/(a+b+c+d) 
detection  (or  recall) 
d/(b+d) 

false  alarms  =  c/(a+c) 
d/(c+d) 

(C.loc  +  D.loc)/ 
(ABCD.Ioc) 

Portland  State 

UNIVERSITY  Results 


•  NBK  will  suffice  (in  85%  cases  NBK  same  or  better  than  J  48) 

•  Early  plateaus  (50-200  examples  are  enough) 

•  Not  shown:  low  PFs 

•  Stratification  improves  PD? 
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level 


Suggestive,  not  conclusive  evidence 
for  "stratification  improves  PD” 
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But,  so  what? 


Is  any  of  the  above  useful? 
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Introducing  -  Process  Simulation 

•  One  area  that  can  help  companies  improve 
their  processes  is  Process  Simulation. 

•  Process  Simulation  supports  organizations 
to  address 

-Strategic  management 

-  Process  Planning 

-Control  and  operational  management 
-Technology  adoption 

-  Understanding 
-Training  and  learning 

-Quantitative  process  management  and  other 
CMMI-Based  Process  Improvement 
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Features  of  Process  Simulation  and  PTAM 

*  Based  on  extensive  research. 

*  Graphical  user  interface  and  models  software 
processes 

*  Utilizes  SEI  methods  to  define  SW  Processes 

*  Integrates  metrics  related  to  cost,  quality,  and 
schedule  into  understandable  performance  picture. 

*  Predicts  project-level  impacts  of  process 
improvements  in  terms  of  cost,  quality  and  cycle  time 

*  Support  business  case  analysis  of  process  decisions 
-  ROI,  NPV  and  quantitatively  assessing  risk. 

*  Designed  for  Rapid  Deployment 
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Importance/Benefits  -  Enduring  Needs 
•  NASA  Project  Level 

-Software  Quality  Assurance  Strategy 
Evaluation  for  NASA  Projects 

-  Independent  Bottoms-Up  NASA  Project  Cost 
Estimation  (Going  where  COCOMO  cannot  - 
KSC  project) 

-  NASA  Contractor  Bid  Evaluation  (NASA  IV&V 
integrated  part  of  Planning  and  Scoping/Cost 
Estimation  strategy) 

-Software  Assurance  Replanning 

-Cost/Benefit  Evaluation  of  new  technologies 
and  tools 
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How  it  works 


Software  Development  Process 
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Better 
Process 
Decisions 

Process  Performance 

Cost,  Quality,  Schedule 


SW  Process 
Simulation  Model 
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Goal 

•  In  this  presentation,  we  assess  the  impact  of  a 
new  technology  (i.e.  Learned  Defect  Detectors) 
on  a  “typical”  large-scale  NASA  project  in  terms 
of  overall  cost,  quality  and  schedule  performance 

•  Goal:  To  determine  when  the  new  technology 
might  be  useful  and  when  they  might  be  useless 
by  providing  a  business  case  to  support  the 
adoption  of  these  tools. 
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Business  Case  Questions 


•  What  is  the  impact  of  applying  new  tools  and  technologies? 

•  What  is  the  economic  benefit  or  value  of  the  tool  or 
technology?  What  is  the  Return  on  Investment? 

•  Under  what  conditions  does  the  tool  or  technology  perform 
best?  Under  what  conditions  does  it  perform  poorly? 

•  What  performance  standards  does  the  tool  need  to  achieve 
in  order  to  have  a  positive  performance  impact  on  the 
project/organization? 

•  Are  there  alternative  ways  to  apply  the  tool  or  technology 
that  enable  it  to  provide  a  more  positive  impact? 
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NASA  Model  -  Includes  IV&V  Layer  with 
IEEE  12207  Software  Development  Lifecycle 
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14 

IV&V  Layer  -  Select  Criticality  Levels  for 
IV&V  Techniques  using  pull-down  menus 


A  Notebook  -  IEEE  12207  PATT2.mox  ^i@ji 
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Assumptions 


£ _ l 


30  ds  ts  cto  ra,  sorts  d  by  s  f fo  rt 


Project  Size  is  100  KSLOC. 

Software  process  follows  the  IEEE  12207+IV&V 
model.  True  for  many  DoD  and  NASA  projects. 

%LOC  lnspected=PD+5%  to  10%;  and 
%LOC  is  proportional  to  Effort 

PF  =  10%-30%. 

PD=40  to  70%. 

The  PD  rate  assumes,  in  turn,  that  defect 
detectors  are  learned  from  data  divided  below  the 
sub-system  level. 

Standard  manual  inspections  find  40%  to  60%  of 
the  total  defects. 

Perspective  Based  inspections  find  80%  to  90%  of 
latent  defects 

Defects  uniformly  distributed  throughout  code 
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Scenario  I  -  Applying  LDD  to  V&V 

•  Learned  defect  detectors  are  applied  during 
project  V&V. 

-  Inspections  are  conducted  on  1 1 .5%  of  code  to  learn 
defect  detectors 

-LDDs  then  applied  to  remaining  code  to  identify  high- 
risk  portions  of  the  system 

-  Explored  the  impact  of  using  higher  PD  combined 
with  higher  PF 

-  Explored  the  impact  of  using  regular 
inspections(weak  training  set)  vs  Perspective  Based 
inspections  (strong  training  set)  for  LDDs. 
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Changes  to  the  Process 
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Scenario  I  -  Results  Summary 


•  Model  recommendations  for  specific  scenarios 

•  General  Rule: 

Insp  Effect*  %Code_lnspected*95%<=  E_LDD*  TS_IE 
Where: 

Insp  Effect  -  Probability  of  detection  of  V&V  inspections 
%Code_lnspected  -  %  of  code  inspected  during  V&V 
E_LDD  -  Probability  of  Detection  for  LDDs 
TS_IE  -  Probability  detection  of  Training  Set  inspections 
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Scenario  I  -  Results  Summary 

•  LDDs  are  Useful  (Significant  benefits)  in  a 
V&V  setting  when: 

-53%  or  less  of  the  code  is  inspected  during  V&V 
(manned  vs  unmanned  missions)  using  regular 
inspections  and  LDD  PD  =50% 

-  Using  high  PD  mode  and  Perspective  based 
inspections 

-  Project  inspections  are  poor 

•  Applying  LDDs  to  V&V  are  Useless  when: 

-  Project  inspections  are  good  or  high  quality 

-  More  than  53%  of  the  code  is  inspected  by  V&V 
(typical  for  manned  missions) 
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Scenario  II  -  Applying  LDD  to  IV&V 

•  Learned  Defect  Detectors  (LDD)  applied  to 
IV&V  (Shedding  light  on  blind  spots) 

-  Project  generated  training  sets  (regular  inspections) 

-  Investigated  the  Impact  of  applying  LDD  to  different 
project  types  (varied  amount  of  code  that  is 
reinspected  (100%-25%» 

-Varied  the  effectiveness  of  reinspection  (2%-10%) 
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Changes  to  the  Process  -  IV&V 
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Scenario  II  -  Results 


•  Clear  recommendations  for  specific  scenarios 

•  Results  (Excellent  Application): 

-  Low  Risk  =  1 .2  PM  with  no  defects  detected 

-  Improves  quality  if  any  defects  are  found  (detection 
capability  >  0) 

-  Receive  added  assurance  even  if  detection 
capability  is  0 

-For  Manned  Missions,  (100%  reinspection),  break¬ 
even  on  total  project  effort  if  IV&V  reinspection 
effectiveness  =  2% 

-  Significantly  improves  cost,  quality  and  schedule  if 
reinspection  effectiveness  is  >=  5% 
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Scenario  II  -  Results 

•  Significant  up  side  potential  when  LDDs 
are  used  to  identify  high  risk  portions  of 
the  code  that  were  not  previously 
inspected  during  project  level  V&V 
(unmanned  missions). 

•  At  50%  code  inspected  by  V&V,  4%-7.5% 
reduction  in  delivered  defects 

•  At  25%  code  inspected  during  V&V, 
reductions  in  delivered  defects  range  from 
15%-24%.  Effort  savings  range  from  18 
PMs  to  29  PMs. 
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Conclusions 

•  Learned  Defect  Detectors  are  useful  when  they 
increase  the  overall  detection  capability  of  the  Coding 
phase. 

•  General  Rule: 

•  Insp  Effect*  %Code_lnspected*95%<=  E_LDD*  TS_IE 

•  This  occurs  when: 

-  Less  than  53%  of  code  is  inspected  during  V&V  or 
V&V  has  week  inspections 

-  Used  as  IV&V  technique  identifying  blind  spots  and 
augmenting  regular  high-quality  V&V 

-V&V  has  weak  inspections 


Portland  State 

UNIVERSITY 

^ ■ 

Conclusions 

•  Learned  Defect  Detectors  are  useless 
when  they  decrease  the  overall  detection 
capability  of  the  Coding  phase. 

•  This  occurs  when: 

-  Used  to  frivolously  cut  costs  by  replacing  high 
quality  code  inspections. 
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Conclusions  -  Broader  Impacts 

•  Identify  the  conditions  under  which 
application  of  a  new  technology  would  be 
beneficial  and  when  applying  this 
technology  would  not  be  beneficial. 

•  We  can  define  performance 
benchmarks  that  a  new  tool  or 
technology  needs  to  achieve. 


Portland  State 

UNIVERSITY 

Conclusions  -  Broader  Impacts 

•  We  can  diagnose  problems  associated 
with  implementing  a  new  tool  or 
technology  and  identify  new  ways  to 
apply  the  technology  to  the  benefit  of  the 
organization  (and  the  vendors) 

•  Finally,  we  can  do  all  this  before  the 
technology  is  purchased  or  applied  and 
therefore  can  save  scarce  resources 
available  for  process  improvement. 


Portland  State 

UNIVERSITY 
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Rolf  W.  Reitzig 

cognence,c 

Integrated  Software  Engineering 


©2002  cognence,  inc. 


Successful  Businesses... 


•  Run  operations  as  if  they  were  a  franchise 

-  Every  business  process  is  standardized 

-  Average  employees  can  easily  be  successful  by  following  the 
processes  as  outlined 

-  Everyone  knows  how  to  perform  their  job 

-  Tasks  are  performed  similarly  on  a  repeatable  basis 

•  A  quality  process  will  yield  a  quality  product 
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Questions  to  Ask... 


•  What  is  the  quality  of  your  delivered  software? 

•  Are  your  software  projects  meeting  estimated  delivery 
dates? 

•  What  visibility  do  you  have  into  your  software  projects? 

•  What  confidence  do  you  have  in  delivering  a  quality 
software  system  on  time  on  budget? 

•  How  do  you  know  if  your  software  projects  are 
performing  well? 

•  What  is  your  cost  of  software  quality? 
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What  is  Software  Process  Improvement  (SPI) 


•  A  planned,  managed,  and  controlled  effort  aimed  at 
improving  an  organization’s  software  development 
capability 

•  Is  most  effective  when  coupled  with  Organizational 
T ransformation  best  practices 
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Return  On  Investment 


•  Organizations  typically  invest  2%-4%  of  their  IT  budget 
on  software  process  improvement 

•  Organizations  engaged  in  a  software  process 
improvement  effort  experience  50%+  gains  in 
productivity  and  a  25%+  decreases  in  post-release 
defects 

•  Average  ROI  was  5:1 

•  Example:  An  IT  department  with  a  S100M  budget 
spending  $4M  on  SPI  can  expect  a  $20M  gain  in 
productivity  over  2  years 
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The  6  Basic  Principles  of  SPI 


1.  Major  changes  to  the  software  process  must  start  at 
the  top 

2.  Effective  change  requires  a  goal  and  knowledge  of  the 
current  process 

3.  Software  process  improvement  requires  investment 

4.  Ultimately,  everyone  must  be  involved 

5.  Software  process  changes  will  not  be  retained  without 
conscious  effort  and  periodic  reinforcement 

6.  Change  is  continuous 

Source:  Humphrey,  W.S.  Managing  the  Software  Process.  Addison-Wesley,  1989 
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Other  Key  Concepts 


1.  To  improve  the  software  process,  someone  must  work 
on  it 

2.  Unplanned  process  improvement  is  wishful  thinking 

3.  Automation  of  a  poorly  defined  process  will  produce 
poorly  defined  results 

4.  Improvements  should  be  made  in  small,  tested  steps 

5.  Train,  train,  train! 


Source:  Humphrey,  W.S.  Managing  the  Software  Process.  Addison-Wesley,  1989 
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Organizational  Transformation 


•  Software  process  improvement  models  build  on 
organizational  transformation  theory  to  ensure 
effectiveness. 

•  Thus,  it  is  imperative  to  understand  organizational 
transformation  theory  in  order  to  improve  the  results  of 
any  software  process  improvement  effort. 
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Organizational  Transformation  Models 


•  Initiating,  Diagnosing,  Establishing,  Acting,  Leveraging 
(IDEAL) 

•  Unfreeze,  move,  refreeze 

•  Envisioning,  Encoding,  Enacting 

•  Awareness,  Understanding,  Definition,  Installation, 
Adoption,  Institutionalization 

•  John  Kotter’s  Model  of  Organizational  Transformation 
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John  P.  Kotter’s  Transformation  Best  Practices 


1.  Establish  a  sense  of  urgency 

2.  Create  the  guiding  coalition 

3.  Develop  a  vision  and  strategy 

4.  Communicate  the  change  vision 

5.  Empower  employees  for  broad-based  action 

6.  Generate  short-term  wins 

7.  Consolidate  gains  and  produce  more  change 

8.  Anchor  new  approaches  in  the  culture 


Source:  John  P.  Kotter,  Leading  Change,  Harvard  Business  School  Press,  1996 
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1  -  Establishing  a  Sense  of  Urgency 


•  Progression  to  subsequent  organizational 
transformation  phases  is  difficult,  if  not  impossible, 
unless  most  managers  honestly  believe  that  the  status 
quo  is  unacceptable 
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2  -  Creating  the  Guiding  Coalition 


•  Successful  transformations  must  be  guided  by  a 
powerful  coalition  that  can  act  as  a  team 

•  The  coalition  is  needed  because  no  one  individual  has 
the  information  needed  to  make  all  major  decisions  or 
the  time  and  credibility  needed  to  convince  lots  of 
people  to  implement  the  decisions 
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3  -  Developing  a  Vision  and  Strategy 


•  Vision  refers  to  a  picture  of  the  future  with  some  implicit 
or  explicit  commentary  on  why  people  should  strive  to 
create  that  future. 

•  3  purposes 

-  Clarifies  the  general  direction  for  change 

-  Motivates  people  to  take  action 

-  Coordinates  the  efforts  of  different  people 

•  Must  be  conveyable  in  5  minutes  or  less 


10/24/2005  cognence  ^  page  13 

©2002  cognence,  inc.  Integrated  Software  Engineering 


4  -  Communicating  the  Change  Vision 


•  The  real  power  of  a  vision  is  unleashed  when  most  of 
those  involved  in  an  enterprise  have  a  common 
understanding  of  its  goals  and  direction 

•  You  cannot  undercommunicate  the  vision! 

•  A  common  mistake  by  the  guiding  coalition  is  to  assume 
the  organization  can  quickly  come  to  grips  with  the 
vision 
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5  -  Empowering  Employees  for  Action 


•  Major  organizational  transformations  rarely  happen 
unless  many  people  assist 

•  Employees  generally  won’t  help  if  they  feel  relatively 
powerless 
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6  -  Generating  Short-Term  Wins 


•  Major  changes  take  time 

•  People  need  to  see  convincing  evidence  that  the  effort 
is  paying  off 

•  Focus  on  short-term  wins  raises  the  urgency  level  and 
ties  the  transformation  effort  to  the  vision  and  strategy 
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7  -  Consolidating  Gains/Creating  More  Change 


•  If  the  sense  of  urgency  is  lowered,  critical  momentum 
can  be  lost  and  regression  follows 

•  Irrational  and  political  resistance  to  change  never  fully 
dissipates 

•  Avoid  the  temptation  to  “take  a  break” 

•  Leadership  must  keep  a  long  term  focus  on  the  vision 
and  anticipated  results 
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8  -  Anchoring  New  Approaches  in  the  Culture 


•  The  goal  is  to  permanently  change  the  organization’s 
shared  values 

•  Cultural  changes  come  last,  not  first 

•  Cultural  norms  are  many  times  difficult  to  change 

•  Cultural  shared  values  are  extremely  difficult  to  change 

•  Will  the  transformation  effort  transcend  any  particular 
individuals??? 
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How  Do  We  Transform  SW  Engineering? 


•  Create  an  infrastructure  that: 

-  Leverages  organizational  transformation  principles 

-  Allows  for  senior  management  prioritization  of  software 
development  process  enhancements 

-  Facilitates  organizational  buy-in  and  cooperation 

-  Encourages  cross-organizational  communication 

-  Reduces  resistance  through  a  reward  system  based  on 
independently  verifiable  achievement  of  management’s 
expectations 

-  Allows  management  visibility  into  the  use  of  standard 
software  development  processes 
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Organizational  Transformation  Infrastructure 


Executive  Vision, 
Direction,  Priorities 


Management 
Steering  Group 


Organizational 
Implementation, 
Standards,  Training 


Engineering 
Process  Group 


Projects  Implement 
Organizational  Standards 


i - 1 - 1 - 1 - ; 


Project  1  ■  Project  2  ■  Project  3  1  ...  ■  Project  n 


l  \  4  l  i 


I 


Quality 

Assurance 


Verifies  Implementation  and 
Adherence  to  Standards, 


Reports  to  MSG,  EPG 
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Roadmap  -  Setting  the  Stage 


1.  Establish  Executive  Sponsorship  with  the  expectation  it  is 
active,  not  passive 

2.  Clearly  tie  the  improvement  effort  to  business  goals 

3.  Establish  a  guiding  coalition  (MSG/EPG)  of  movers  and 
shakers  from  across  the  organization  to  drive  the  strategy, 
approach,  and  plan 

4.  Projectize  the  effort,  assign  a  cost  center,  and  treat  it  like  a 
project  with  clear  milestones  and  reviews 

5.  Conduct  a  comprehensive  process,  project,  personnel,  and 
financial  appraisals  to  establish  an  organizational  baseline 

6.  Tie  process  improvement  objectives  to  each  individual’s 
performance  review 
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Roadmap  -  Introducing  Improvements 


7.  Establish  a  measurement  capability  early,  but  don’t 
overwhelm  projects  with  data  gathering  requirements 

8.  Establish  QA  early  to  help  guide  and  mentor,  and  to  report 
process  improvement-related  progress 

9.  Ensure  project  schedules  going  forward  contain  all  the 
required  elements  to  meet  your  CM  Ml  improvement  objectives 

10.  Either  adopt  processes  (that  meets  your  needs!),  or  have  the 
EPG  design  ones  that  are  better  suited 

11.  Projects  execute  CMMI-compliant  processes  and  begin 
performing  better! 

12.  Continue  to  monitor  key  business  measures,  execute  QA,  and 
conduct  senior  management  reviews  to  drive  urgency. 
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End  Result 


•  The  outcome  will  be  an  integrated,  organizationally 
cooperative  infrastructure  that: 

-  is  the  foundation  for  a  successful  organizational 
transformation 

-  facilitates  software  process  improvement  based  on 
consensus  priorities 

-  provides  an  environment  that  supports  project  buy-in  and 
adoption  of  improvements 

-  communicates  effectively  across  the  organization 

-  reports  results  to  senior  management 
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Questions? 


cognencemc 

Integrated  Software  Engineering 
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Steve  Ross  and  Kurt  McMillen 

November  2005 
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Organization  and  Accomplishments 


Raytheon 

Customer  Success  Is  Our  Mission 


Raytheon  Missile  Systems,  Tucson,  AZ 


Employees:  11,000 


’04  Sales:  $3.8  B 


Worlds  Largest  Appraised  SEI 
CMMI  Level  3  Organization 
December  2004 


SW-CMM  Level  5  in 
November  2001 
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The  Case  for  Change: 

Our  “Burning  Platform”  Was  Ablaze  . . . 


Raytheon 

Customer  Success  Is  Our  Mission 


December  2003  Class  2  Appraisal  -  A  long  way  to  go! 


And  we  were  the  remaining  RMS  business  unit  not  appraised  to 

CMMI  Level  3 . . . 
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And  The  Pressure  Was  On  . . . 


Raytheon 

Customer  Success  Is  Our  Mission 


Missile  Systems 

Business  Goals 


Customer  Satisfaction 


*  Know  our  customers  and 
understand  their  needs 

■  Always  perform  to  our 
commitments 

*  Grow  strong  trusting 
relationships  at  alt 
customer  levels 

*  Be  accountable  for  the 
entire  product  life  cycle 


Be  a  Customer-Focused 
Business 


Engage  our  customers  and 
suppliers  in  partnerships  for 
success 

Fulfill  evolving  Warfighter 
requirements  with  innovative 
technology,  products  and 
solutions 

Leverage  company- wide 
capabilities 

Grow  programs  and  presence 
worldwide 

Create  new  business 
opportunities  from 
innovative  ideas 


Provide  Superior  Solutions 
for  the  Warfighter 


People 


*  Ensure  a  respectful,  productive 
and  safe  environment 

1  Improve  our  culture  by 
valuing  diversity 

*  Develop  and  recognize  people 

►  Take  personal  responsibility 
for  open  and  honest 
communication 


Help  Each  Other  Succeed 


Productivity 


•  Maintain  an  unrelenting 
focus  on  affordability  and 
cycle  time  in  everything 
we  design  and  produce 

►  Drive  speed  and  agility 
through  R6a  IPDS, 
Commonality  and 
Process  Improvement 

►  Meet  financial  goals 

►  Drive  Supply  Chain 

Achieve  CMMI  Level  3 


Drive  Performance 
Excellence 


Customer  Success  Is  Our  Mission 


Raytheon 

Missile  Systems 
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Raytheon 

Customer  Success  Is  Our  Mission 


Q:  If  IPDS  @  RMS  is  our  set  of  best  practices  for 
planning  and  executing  complex  missile  programs, 
why  aren’t  we  using  it  to  plan  and  execute  our  CMMI 

Level  3  Project? 

We  had  the  guidance  in  place  (a  subset): 

•  Creation  of  IPDS  @  RMS  Tailoring 

•  Program  Leadership,  Planning  and  Support 

•  Program  Resourcing,  Financial  Planning  and  Management 

•  Program  Monitoring  and  Control 

•  Gate  Independent  and  Start-Up  Reviews 

•  Supporting  Processes 

•  Configuration  and  Data  Management,  Measurement  and  Analysis,  Objective 
Evaluation,  Peer  Review,  Risk  Management 

•  The  “Organizational  Level  Stuff” 

•  Organizational  Process  Focus,  Organizational  Training  . . . 


The  Epiphany 


It  is  NOT  “ rocket  science”;  every  type  of  project  should  be  able  to 

leverage  the  defined  enterprise  process 
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A  Short  Course  on  IPDS  @  RMS:  Raymeon 

The  Top-Level  View 

«  * 


Rayttwon 

Sum*#  ^  Wj'hw 

IPDS  @  RMS  I  Storyboard  I  Index  I  Process  Asset  Library  I  Tailoring  Tool  I  RMS  CMMi 

CM  Ml©  RMS  IPDS  @  RMS  Version  1.5 

Corporate  IPDS 


IPDS  @  RMS 


Raytheon  Rome  |  Dir^cloiy  Search  |  HettiJacmr  |  Col  I  abu  fa  C  i  On  \  Help' 


IPDS  ©RMS 
Feedbackj'CR  Tool 

IPDS  ©  RMS  Contacts 

Ml  sale  Systems  Home 


IPDS  @  RMS  Quick  Link s 


.'PDS  i.'  fli VIS  Pmi/jwj 


Submit  gn  IPDS  i?  PfJW'S 
CtiBngB  ffocjycrsr 


IPOS  ®  RMS 
Compononls 


Storyboard 
Storyboard  Index 


Welcome  to  IPDS  @  RMSr  your  home  base  for  Missile  Systems '  implementation  of 
the  Raytheon  Integrated  Product  Development  System.  Inside  you  will  find  the  core 
process  tasks ,  methods  and  enablers  you  need  to  achieve  your  results  in  business 
development,  program  management  and  product  development.  Process 
simplification,  integration  and  ease-of-use  are  the  IPDS  @  RMS  priorities . 


IPDS  @  RMS  PAl 


Program  Tailoring, 
IPDS  @  RMS  Gating 
PlDL  Home 


IPDS  @  RMS  News 


09/05  What's  New  In  IPDS  @  RMS  Version  1,5 

Archives 


Previous  and 
Developing  Versions 
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A  Short  Course  on  IPDS  @  RMS:  Raytheon 

Our  Process  Architecture  a-— 

«  * 


* 


Raytheon 


Missile  Systems  Nome 


Riiythfrun  Home  f  Directory  |  Search  I  HdMJnorii  |  Calhharalicm  |  H  c  Ip 


IPDS  @  RMS  Story board 

Version  1.5 


MS  leadership  I  MS  Organizations 


CommuniiiGS  Employee  Services 


Resources 


IPOS  @  RMS  Storyboard  Home 


IPDS  RMS  I  Glossary  I  Tailoring  I  Policies  I  TO  Index  I  Methods  /  Enablers  I  Help 


Business  5  tratc  gy  Project  Plann  in  g,  Roqu  KTEtn  ents  & 

Planning/Execution  Mgl.  &  Control  Architecture  Dev. 


Engineering 


Product  Design 
and  Development 


System 
Integration" 
Verification  & 
Validation 


Production  and 
Deployment 


Operations  & 
Support 


Returning  Users 

Se/ec t  your  Stage  from 
the  A/a  vigation  Bar 
Ah  ore 

New  Users 


IMhaf 1  £  M&w  jr  IPDS  &  RMS 


•>& 


yy 


iPDS  fci  RMS  Ptttuiouf  hferfigns 


Submil  ait  IPDS  «1  f?JUS 
Change  Reg  nest 


Welcome  to  the  IPDS  @  RMS  Story  board f  your  home  base  for  Missile  Systems ' 
implementation  of  the  Raytheon  Integrated  Product  Development  System.  Inside 
you  will  find  the  core  process  tasks r  methods  and  enablers  you  need  to  achieve 
your  results  in  business  development ,  program  management  and  product 
development.  Process  simplification ,  integration  and  ease-of-use  are  the  IPDS  @ 
RMS  priorities. 


IPDS  @  RMS  News 


09/05  What's  New  In  IPDS  @  RMS  Version  1 .5 

Archives 


Copyright  ©  2005  Raytheon  Company.  All  rights  reserved. 


Page  7 


A  Short  Course  on  IPDS  @  RMS:  Raymeon 

Our  Tailoring  Guidance 

«  * 


Raytheon 

IPDS  @  RMS 

Cvftow  iiKWifitOvrmm  uaytiMon  home  | 

Dir?  cl  ary  SSarnh  |  TJewSiunm  |  CutlabftTirClaFl  | 

kPDS  @  RMS  1  Storyboard 

Index  1  Process  Asset  Library 

Tailoring  Tool 

RMS  CMMI 

CM  Ml  @  RMS 
Corporate  IPDS 


IPDS  @  RMS  Program  Tailoring 


IPDS  iO!  RMS 
Feetfbacki'CR  Took 

IPDS  ®  RMS  Contacts 

Mi  sale  Systems  Home 


IPDS  RMS 
Compel  nents 


Story  board 
Storyboard  Index 


The  purpose  of  IPDS  @  RMS  Tailoring  is  to  particularize 
the  common  processes  defined  in  IPDS  @  RMS  to  a 
specific  program  instantiation  that  meets  the  program's 
unique  needs.  The  tailoring  activity  results  in  an  initial 
program  IMP/IMS  and  information  can  be  included  in  other 
program  plans.  An  integrated,  web  based  Tailoring  Tool  is 
provided  to  allow  program  managers  and  planners  to  easily 
tailor  IPDS  @  RMS  tasks  and  methods  to  create  program 
planning  structures.  The  tailoring  tool  readily  exports  the 


IPDS®  RMS  PAL 


Program  Tailoring 
Tailoring  Tool 
Tailoring  Tool  User  Guido 


Program  Planning 
Playback 
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Our  CMMI  L3  Project  Architecture:  Raytheon 

The  Appraisal  Objectives  and  Scope 

♦  Objectives 

•  Review  IPDS  @  RMS  process  content  and  its 
implementation  against  the  CMMI-SE/SW  model 
VI  .1  (Staged  Representation) 

•  Identify  process  and  deployment  weaknesses 

•  Measure  performance  against  CMMI  Level  3 

♦  Organization  Being  Appraised 

•  Raytheon  Missile  Systems 

♦  CMMI  Scope 

•  All  CMMI  Level  2  and  Level  3  Process  Areas 

A  daunting  task  based  upon  the  complexity  of  the  RMS 

organization 


Copyright  ©  2005  Raytheon  Company.  All  rights  reserved. 


Page  9 


So  How  Do  We  Get  There  From  Here? 
Our  Roadmap 


Raytheon 

Customer  Success  Is  Our  Mission 


♦  Tailor  IPDS  @  RMS 

♦  Plan  the  project 

•  Establish  /  prioritize  tasks  and  their  relationships 

•  Identify  and  assign  resources,  roles,  responsibilities  and  authority 

•  Train  the  team  and  clearly  communicate  required  work  products 
and  criteria  for  success 

♦  Project  start-up  review  (Gate  5  IR  and  Review) 

♦  Manage  execution  of  plans  and  track  progress  against  plan 

♦  Manage  risks  and  take  action  as  appropriate 

♦  Objectively  evaluate  adherence  to  product  &  process 
requirements 

♦  Review  status  with  upper  management 

♦  Identify  improvement  opportunities 

A  very  typical  project  plan 
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CMMI  Level  3  Project  Tailoring: 
Our  Concept  of  Operations 


Raytheon 

Riylttfi-Oifii  Home  |  Directory  |  Search  |  News  roam  |  Col  I  aba  rati  an  |  Help 
RMS  CMMI  I  IPDS  &  RMS  I  Storyboard 


IPDS  @  RMS  Tailoring  Tool 

Tailoring  Report 


TAILORING  TOOL  HOME  PROGRAM  LIST  ADD  PROGRAM  ADMINISTRATION  LOGOUT 


Raytheon 

Customer  Success  Is  Our  Mission 


IPDS  @  RMS  Tailoring  Tool 


REPORT  FILTERS 

Process  Area:  |NONE 

Method  Owner  Group:  |NONE 

SEARCH  | 


3 


Select  a  Process  Area  (or  ALL)  to  view 
tasks  within  the  Tailoring  Report. 

Select  a  Method  Owner  Group  (or  ALL) 
to  view  methods  within  the  Tailoring 
Report. 


PROGRAM  DETAILS 

Program  name: 

IPDS  @  RMS  Version  Nbr: 
Product  Line: 

Start: 

Finish: 

Release  to  Public: 

Status: 

Program  Description: 


Program  Assumptions: 


RMS  Organizational  Tailoring  (EPO  and  Functional) 

1.2 

Productivity 

04/29)04 

04/28/06 

Y 

Approved 

This  document  contains  the  organizational-level  tailoring  of  IPDS  @  RMS  for  the  Enterprise,  Systems  Engineering  (SE)  and 
Software  Engineering  (SW).  The  accepted  and  modified  activities  will  be  reflected  in  the  organizational-level  plans:  1 .  IPDS  @ 
RMS  System  Manual  2.  IPDS  @  RMS  Strategic  Process  Improvement  Planning  document  3.  IPDS  @  RMS  Annual  Process 
Improvement  Plan  4.  IPDS  @  RMS  Measurement  &  Analysis  Plan  5.  IPDS  @  RMS  Configuration  Management  Operating  Plan  6.  All 
other  plans  as  required  at  the  Enterprise  or  SE/SW  functional  levels. 

Pre-Processing  (Stage  /  Gate  /  PO/  MO)  Conditions:  1 .  At  the  Process  Owner  level,  Stage  2  and  Gate  5  apply  for  this  activity. 
Pre-Oates  6  through  10  activities,  though,  will  be  accomplished  that  are  applicable  to  the  organizational-level  process  definition 
and  improvment  management.  Gate  11  is  not  included  due  to  the  constrained  two-year  timeframe  applicable  to  this  tailoring.  2. 
At  the  Supporting  Process  Owner  level,  the  following  apply:  OPD,  OPF,  OEI,  OT,  CM/DM,  DAR,  MA,  Peer,  Quality  /OE  and  SOM. 
3.  At  the  Method  Owner  level,  the  following  apply:  PM,  Quality,  CM/DM,  OPD,  OPF,  OEI,  SOM  and  Training. 


We  were  able  to  perform  significant  “ pre-processing ” 
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The  Tailoring  and  Planning  Result:  Raymeon 

Our  Organizational  “Plan  Tree” 


IPDS  @  RMS  Org  Process 
Mgmt  Plan  (OPMP) 


IPDS  @  RMS  System  Manual 


X 


IPDS  @  RMS  Strategic 
Process  Impr  Planning  (SPIP) 


IPDS  @  RMS  M&A  Plan 


SE  M&A  Plan 


SW  M&A  Plan 


IPDS  @  RMS  Tng  Plan 


SE  Tng  Plan 


SW  Tng  Plan 


IPDS  @  RMS  CMOP 


I  IPDS  @  RMS  OE  Plan  <- 


IPDS  @  RMS  Risk  Plan  <- 


IPDS  @  RMS  Annual  PIP 


SE  APIP 


SW  APIP 


And  we  executed  to  our  plans 
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The  Tailoring  and  Planning  Result: 
Our  Project  Master  Schedule 


Raytheon 

Customer  Success  Is  Our  Mission 


2004 


1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

J 

F 

M 

A 

M 

J 

J 

A 

S 

O 

N 

D 

1 .0  RMS  Reviews 

RMS  Process  Quarterly  Reviews 
Product  Line  Reviews 
Staff  Meetings 

2.0  Management 


Front-Runner  Meetings 


Gate  5 

♦  ♦ 
♦  ♦  ♦ 

Bi-weekly 


♦ 

♦ 


♦  ♦ 


♦ 

♦ 


3.0  Appraisals 


♦  Front-Runner  Intro 

♦  ♦  ♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦<><> 

i 

Every  working  Friday 


♦ 

♦ 

♦ 

♦ 

♦ 

♦40 

IR  1 

Mini 

PBA 

IR  2 

PBA 

IR  3 

SCAMPI 

13-15 

3-14 

7-11 

26  July 

Sep  27  - 

1 5-l|9  Nov 

APR 

May 

June 

-6  Aug 

Oct  22 

6-1  y  Dec 

4.0  Deployment 

Program,  Org  Increment 


5.0  Infrastructure 


IPDS  @  RMS 


Update  ♦ 


♦ 

Release 

VI  .2 


Update  ♦♦ 

Release 

VI  .3 


A  very  typical  project  master  schedule 


Copyright  ©  2005  Raytheon  Company.  All  rights  reserved. 


Page  13 


Just  Doing  An  Activity  Is  Not  Enough:  Raymeon 

■■  I  ■  ■  m  u  m  Customer  Success  is  Our  Mission 

Our  Supporting  Deployment  Infrastructure 


To  Deploy  a  Process  Instance 

Tasks 

Reference 

Process 

GP 

Input  Process 

Process  X 

3.1 

Tailor 

Process 

3.1 

Plan 

Process 

2.2  + 

Resource 

Process 

2.3,  4 

Train 

Process 

2.5 

Execute 

Process 

3.1 

QA/Peer  Review 

Process 

2.9 

CM 

Process 

2.6 

Mon/Control 

Process 

2.8,  2.7 

Review  w/  Mgt 

Process 

2.10 

Collect  Artifacts 

Process 

3.2 

t 


j_  i 


L_ L_i  Hi 

i_  L  i_  i_U. 


1 


1 


I  I  I  III  III 


Proper  deployment  is  key  to  successfully  changing  the  culture 
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Raytheon 

The  RMS  CMMI  Level  3  Project  Team  c 

«  w 


And  the  CMMI  Level  3  Project  was  one  of  the  Process  Action 

Teams  (PATs)  under  the  EPG 


Copyright  ©  2005  Raytheon  Company.  All  rights  reserved. 


Page  15 


Raytheon 

Training  Strategy  (An  Extract)  ““ 


IPDS  @  RMS  Training  Matrix 

Training  Module 

Content 

Process  Owner  and  Method  Owner  Reps 

How  Core  Process,  Core  Support  Process,  Methods  and  Enablers  Owners  input  new  and  updated  content  into  IPDS  @  RMS. 

System  Admin  Training 

Access  control  and  application  administration 

Program  Planning  Team 

IPDS  @  RMS  program  startup  (from  initial  WBS  development  through  tailoring  to  IMP/IMS  generation)  based  upon  program 
attributes  and  business  needs. 

Senior  Management 

Business  applicability  of  IPDS  @  RMS;  Senior  management  roles  and  responsibilities  with  respect  to  IPDS  @  RMS. 

IPDS  @  RMS  Overview 

IPDS  @  RMS  development  history  and  architecture;  includes  information  about  how  the  individual  contributor  navigates  through 
and  uses  IPDS  @  RMS. 

IPDS  @  RMS  Tailoring  Training 

Tailoring  goals,  objectives,  and  methodology 

EPG  Roles  &  Responsibilities 

How  the  organization  defines,  manages,  deploys  and  improves  IPDS  @  RMS 

Appraisal  Training 

Preparing  for  our  CMMI  assessment 

Change  Control  Management 

IPDS  @  RMS  CM  responsibilities 

IPDS  @  RMS  OE  PPQA  Training 

Objective  Evaluation  for  Process  and  Product  Quality  Practitioners 

IPDS  @  RMS  CMMI  Overview 

Provides  the  background  familiarization  of  CMMI,  the  language  used,  and  how  it  affects  IPDS  @  RMS 

Training  modules  were  identified  based  upon  IPDS  @  RMS  use 

cases  across  the  RMS  business 
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Raytheon 

EPG  Training  Matrix  (An  Extract) 


Enterprise  Process  Group  (EPG)  Roles 

Course 
Number [11 

Course  Title 

Course 

Hours 

EPG  Lead 

EPG  Chief  Engineer 

EPG  IPT  Leads 

EPG  Team  Members 

IPDS  @  RMS  CCB 

Members 

Process  Owner 

Reps 

Method  Owner 

Reps 

Deployment 

Team  Members 

Process  Area  Experts 

(PAEs) 

RMS-CMMI01 

RMS  CMMI  Overview 

2 

R 

R 

R 

R 

R 

R 

R 

R 

R 

KCSW001 7 

IPDS  @  RMS  Overview 

2 

R 

R 

R 

R 

R 

R 

R 

R 

R 

KCSW0018 

IPDS  @  RMS  Senior  Management  Training 

2 

R 

S 

KCSW0042 

IPDS  @  RMS  Program  Planning  Team  Training 

2 

S 

S 

S 

A 

A 

R 

A 

CMMICC02 

IPDS  @  RMS  Change  Control  Training 

1 

R 

R 

R 

R 

R 

R 

R 

R 

R 

KCSW0041 

IPDS  @  RMS  EPG  Roles  &  Responsibilities  Training 

2 

R 

R 

R 

R 

S 

R 

R 

S 

S 

QUAL2004 

IPDS  @  RMS  OE  PPQA  Training  (can  be  taken  ILO  QUAL2004) 

4 

S 

S 

S 

S 

S 

S 

S 

S 

QUAL2005 

IPDS  @  RMS  OE  PPQA  Overview  Training 

2 

R 

R 

R 

R 

S 

R 

R 

R 

R 

CMMI201 

Introduction  to  CMMI 

24 

R 

R 

R 

S 

S 

S 

S 

S 

R 

SYS0017 

Risk  Management  Overview  for  Chief  Engineers  and  IPT/Section  Leads 

2 

R 

R 

R 

JCMMI01 

M&A  and  PMC  Overview 

2 

R 

R 

R 

R 

A 

R 

KCSW0053 

IPDS  @  RMS  Tailoring  Overview 

2 

S 

S 

S 

S 

S 

S 

S 

KCSW0043 

SAM  Appraisal  Training  for  Interviewees 

1 

s[4] 

s[4] 

s[4] 

s[4] 

KCSW0046 

Organization  Representative  Appraisal  Training  for  Interviewees 

1 

s[4] 

s[4] 

s[4] 

s[4] 

s[4] 

s[4] 

s[4] 

s[4] 

s[4] 

KCSW0050 

Senior  Management  Appraisal  Training  for  Interviewees 

1 

s[4] 

s[4] 

s[4] 

s[4] 

s[4] 

JAC1123 

Project  Metrics 

8 

R 

R 

R 

R 

A 

R 

LEGEND _ 

R  =  Required  Course 
S  =  Suggested  Course 

A  =  As  Appropriate  (determined  by  individual  and/or  manager) 


And,  like  programs,  we  identified  specific  project  training  by  role 
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What  We  Learned:  Raymeon 

Key  “Knowledge  Gained”  a-mrs****,**, 

♦  Plan  for  success  (GP  2.2) 

•  Leverage  off  of  your  already-defined  best  practices 

•  Appraisal  internal  reviews 

•  CMMI  project  reviews 

♦  Keep  the  end  in  mind  (GP  2.2) 

•  Reverse  planning 

♦  Make  the  plan  visible  to  the  team  (GP  2.2,  GP  2.7) 

•  CMMI  project  plan  on  the  war  room  wall 

•  Stakeholder  meetings  held  where  plan  was  visible 

“ Critical  Chain  ”  methods,  tools  and  techniques  were  extremely 

valuable  enablers  to  our  success 
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What  We  Learned:  Raymeon 

Key  “Knowledge  Gained”  a-mrs****,**, 

♦  Manage  to  the  plan  (GP  2.8) 

•  Daily  stand  up  meetings  focused  on  the  critical  chain 

♦  Engage  all  levels  of  management  (GP  2.4) 

•  The  role  of  our  Program  Director  and  Program  Manager  was 
primarily  to  remove  barriers 

•  Do  not  take  the  responsibility  and  accountability  from  those 
who  are  responsible  and  accountable 

♦  Get  the  right  team  (GP  2.3,  GP  2.4) 

•  Identify  key  skills 

•  Establish  roles  and  responsibilities 

•  Adequately  resource  a  deployment  support  team 
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What  We  Learned:  Raymeon 

Key  “Knowledge  Gained”  a-mrs****,**, 

♦  Ensure  team  members  understand  their  roles,  the 
program  objectives,  and  commit  to  the  plan  (GP  2.7) 

♦  Train  the  team  (GP  2.5) 

•  Model  training 

•  Appraisal  training 

•  IPDS  @  RMS  training 

•  Roles  and  responsibilities  training 

♦  Review  progress  with  senior  management  (GP  2.10) 

•  Sponsor  reviews 

•  Center  manager  breakfasts 

•  Program  management  lunches 


Copyright  ©  2005  Raytheon  Company.  All  rights  reserved. 


Page  20 


What  We  Learned:  Raymeon 

Key  “Knowledge  Gained”  a-mrs****,**, 

♦  Identify  and  manage  the  work  products  (GP  2.6) 

•  Appraisal  evidence 

•  Plans 

•  Presentation  material 

•  Intermediate  work  products 

♦  Make  sure  the  process  is  effective  (GP  2.9) 

•  Objectively  evaluate  the  program  plans  and  processes 

•  Develop  meaningful  and  effective  management  indicators  and 
tracking  tools 

♦  Identify  /  implement  improvements  (GP  3.2) 

•  Center  manager  breakfasts 

•  Program  manager  lunches 

•  Appraisal  internal  reviews 

•  CRs  to  IPDS  @  RMS, . . . 
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The  Proof  Of  Our  Planning: 

Completion  on  Schedule  &  Within  Budget 


Raytheon 

Customer  Success  Is  Our  Mission 


2004 


1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

J 

F 

M 

A 

M 

J 

J 

A 

S 

O 

N 

D 

1 .0  RMS  Reviews 

RMS  Process  Quarterly  Reviews 
Product  Line  Reviews 
Staff  Meetings 

2.0  Management 

Front-Runner  Meetings 

3.0  Appraisals 


Gate  5 

♦  ◄ 

♦  ♦  ◄ 

Bi-weekly 

♦  Front-Runner  Intro 

♦  ♦  ♦♦♦ 

Every  working  Friday 


♦  ♦ 


♦ 

♦ 


oo 


♦ 

♦ 

♦ 

♦ 

♦ 

♦4-0 

IR  1 

Mini 

PBA 

IR  2 

PBA 

IR  3 

SCAMPI 

13-15 

3-14 

7-11 

26  July 

Sep  27  - 

15-1-9  Nov 

APR 

May 

June 

-6  Aug 

Oct  22 

6-1 7  Dec 

4.0  Deployment 

Program,  Org  Increment 


5.0  Infrastructure 


IPDS  @  RMS 


Update  ♦ 


♦ 

Release 

VI  .2 

05  April 


Update 


Release 

VI  .3 
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The  Proof  Of  Our  Planning:  Raymeon 

PBA  Result  (August  2004)  a™,**,**. 

«  w 


Maturity  Level 

Level  2 

Level  3 

Goals 

REQM 

CL 

CL 

PMC 

SAM 

< 

< 

o 

CL 

CL 

o 

O 

DC 

CO 

i— 

CL 

DC 

LU 

> 

VAL 

OPF 

o 

CL 

O 

1— 

O 

IMP 

RSKM 

DC 

< 

O 

GG3 

B 

B 

B 

D 

B 

D 

D 

c 

c 

C 

c 

c 

B 

B 

c 

c 

c 

GG2 

c 

c 

c 

c 

B 

B 

c 

SG4 

SG3 

B 

A 

B 

B 

B 

B 

A 

SG2 

B 

B 

c 

D 

B 

A 

B 

c 

D 

D 

B 

c 

B 

A 

A 

SGI 

B 

c 

c 

B 

B 

A 

A 

B 

B 

B 

B 

B 

B 

c 

B 

c 

A 

B 

August  2004  Class  B  Appraisal  -  Progress  on  all  fronts  on  par 

with  our  plan 
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The  Proof  Of  Our  Planning:  Raymeon 

Level  3  Success  Chart  (December  2004)  c““ceK's0"r‘fei"’ 

«  w 


Maturity  Level 

Level  2 

Level  3 

o 

< 

o 

a 

DC 

_ 1 

LL 

O 

n 

DC 

Goals 

LU 

CL 

< 

< 

CL 

o 

CO 

LU 

< 

CL 

CL 

1— 

1  * 

CO 

< 

DC 

CL 

CL 

CO 

CL 

O 

DC 

1— 

CL 

> 

> 

O 

O 

o 

DC 

O 

GG3 

s 

S 

S 

s 

s 

s 

s 

s 

s 

s 

s 

s 

s 

s 

s 

s 

s 

s 

GG2 

s 

S 

s 

s 

s 

s 

s 

SG4 

SG3 

S 

s 

s 

s 

s 

s 

s 

SG2 

S 

s 

s 

s 

s 

s 

s 

s 

s 

s 

s 

s 

s 

s 

s 

SGI 

s 

s 

s 

s 

s 

s 

s 

s 

s 

s 

s 

s 

s 

s 

s 

s 

s 

s 

December  2004  SCAMPI  -  Success! 
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The  Proof  Of  Our  Planning:  Raytheon 

Other  CMMI  Level  3  Project  Success  Indicators 

♦  Raytheon  Missile  Systems  is  the  single  largest 
organization  to  achieve  CMMI  Level  3 

♦  Schedule  Performance  Index  -1.0 

♦  Cost  Performance  Index  -  1 .02 

♦  Appraisal  success  indicators 

•  Appraisal  was  not  a  discovery  process 

•  No  late  nights  on  the  appraisal  team 

•  No  weekend  work 

♦  Significant  Findings:  0 

♦  Weaknesses:  4 

♦  Opportunities  for  Improvement:  3 

♦  Strengths:  8 
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Raytheon 

Customer  Success  Is  Our  Mission 


Questions? 

Thank  you  for  your  participation ! 
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Seer 


Summary 


9  Measurement  objectifies  management 
9  Estimation  is  a  function  of  progress  (continuous  process) 

A  well-formed  estimate  is  specified  as  a  probability  distribution 
«  Uncertainty 

•  Variability 

•  Risk 

•  Opportunity 

»  Software  size  estimates  <■ 

•  Size  growth 

•  Size  estimation  variability 
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Software  Development  and 
Measurement 


^  Staffing  ^ 
(people,  time) 

Effort 

(person-months) 

T 

Labor 

Cost 

(currency) 

Size 

(work  units) 

_ 


Desire 


Start 1 


Effective  Technology 

(coefficient) 


Technology 


Friction 


Software 
Development 
Process 


Defects 

(count) 


Software 


Time 

(calendar  months) 


\ Finish 


Size 

(work  units) 
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Fundamental  Measures 


Size 

Effective  Technology 
Time 

Effort  ^  Cost ,  Staffing 
Defects 
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Estimate  Defined 


es*ti*mate  (es'ti  mit),  n. 

an  approximate  judgment  or  calculation ,  as  of  the 
value  or  amount  of  something 

a  prediction  that  is  equally  likely  to  be  above  or  below 

the  actual  result  (Tom  DeMarco) 

A  WELL  FORMED  ESTIMATE 
IS  A  DISTRIBUTION 


Metric  Metric 
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Two  Key  Drivers  of 
Software  Size  Estimates 


o  Size  Growth 

•  Change  in  the  baseline  estimated  software  size  due  to: 

-  Change  in  development  and/or  operating  environment 

-  Change  in  the  required  functionality 

•  Technological  and  Programmatic  risk 
O'  Size  Estimation  Variability 

•  Estimation  process  variability  due  to: 

-  Human  behavior 

-  Model  behavior 
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Size  Growth 


9  Operational  Environment  Volatility 

•  The  mission  changes. 

•  The  regulations  that  govern  how  this  software  should  behave  have 
changed. 

9  Essence  (Requirements)  Volatility 

•  The  customer  doesn’t  know  what  he/she  wants. 

9  Essence  Understanding  (Requirements  Completeness  and 
Correctness) 

•  The  customer  doesn’t  understand  the  problem. 

•  The  specifications  are  vague. 

9  Essence  versus  Implementation  Correspondence 

•  The  vendor  adds  a  few  extra  features  (gold  plating). 
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Growth  Factor  Function 


Q  Yields  Growth  Factor  as  a  function  of  normalized  earned  value 
0  Based  on  Galorath  Incorporated  analysis  of  historical  data 
Q  Embedded  in  SEER-SEM™* s  Phase  at  Estimate  parameter 


G  ( s)  —  —0.7  s  +  0.69 
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Growth  Factor  Function  Distribution 


o  Triangular  Distribution  per  (Book  2002) 
o  Skew  per  modified  (Tarbet  2002) 


G(s)  =  [L  M  tf]  =  [0  0  G(s)] 
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Growth 

Impact  Distribution 


o  Function  of  normalized  earned  value  (progress) 
a  Product  of  best  guess  size  and  growth  factor 
Q  Triangular  Distribution  per  (Book  2002)  from  growth  factor 
0  Skew  per  (Tarbet  2002)  from  growth  factor 


Sg(s)  =  Sm(s)G(s)  =  [0  0  SM(i)G(s)] 
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Size  Growth 
Example  Calculation 


o  Assume  a  best  guess  size  at  SRR  of  50,000  ESLOC 
o  Assume  normalized  earned  value  of  1 1 .8%  at  SRR 


(*SRR 

S! 

°G_SRR 


G  (1 1 .8%)  = -0.7  (1 1 .8%) +0.69 

=  >  0  Sm_jSS(0.61)]  =  [0  0 


=  0.61 
30,500] 
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Probability  Density 
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Size  Growth 
Example  PDF 
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Probability  Density  versus  Software  Size 


Software  Size  (effective  source  statements) 

October  24,  2005 


12 


Confidence  Probability 


Seer 


Size  Growth 
Example  CDF 
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Confidence  Probability  versus  Software  Size 


Software  Size  (effective  source  statements) 
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Size  Estimation  Variability 


9  Uncertainty  about  the  translation  of  essence  to  implementation 
9  Error  and  bias  introduced  by  the  estimation  process 
9  Error  and  bias  introduced  by  the  estimation  model  /  relationships 
9  Error  and  bias  introduced  by  the  people  performing  the  process 
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Size  Estimation  Variability 
impact  Distribution 

o  Function  of  normalized  earned  value  (progress) 

&  Normal  (Gaussian)  Distribution  per  (Book  2002) 

*  Variance  per  (Tarbet  2002) 


[m  cf] 


(30  %)SU 
(2)(2.33) 


[0  3,219] 
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Size  Estimation  Variability 
Example  Calculation 


o  Assume  a  best  guess  size  at  SRR  of  50,000  ESLOC 


EV_SRR 


(30%)  W 

(2)(2.33) 


[0  3,219] 
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Probability  Density 


Seer 


Size  Estimation  Variability 
_ Example  PDF _ 


PDF 

Probability  Density  versus  Software  Size 


Software  Size  (effective  source  statements) 
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Confidence  Probability 


Seer 


Size  Estimation 
Example  CDF 


CDF 

Confidence  Probability  versus  Software  Size 


Software  Size  (effective  source  statements) 
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Combining  Size  Growth  and 

Size  (Estimate)  Uncertainty 


o  The  mean  of  the  sum  of  a  set  of  random  variables  is  equal  to  the  sum  of 
the  means  of  each  random  variable  in  the  set 

Q  The  standard  deviation  of  the  sum  of  a  set  of  independent  random 
variables  is  equal  to  the  square  root  of  the  sum  of  the  squares  of  the 
standard  deviations  of  each  random  variable  in  the  set 


ZE(X() 

1=1 

£v(X() 
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Combining  Size  Growth  and 

Size  (Estimate)  Uncertainty 


o  Sum  of  the  means: 


Sm(5) 

0+0+Sm(j)G(j)  s,(.)c(.| 

^sG(s)  2  2 

^SEY(s) 

.  „  _c  ,,|S,WaW_s1,W(GW+3) 

*  *  Ms(s)  ~  \S)1~  ~  ~  q 
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Combining  Size  Growth  and 

Size  (Estimate)  Uncertainty 


o  Square  root  of  the  sum  of  the  squares  of  the  standard  deviations: 


l2+m2+h2-  lh-lm -mh 


(7. 


(7. 


sg(5)  V  j^g 

(30%)  SM  ( 5 ) 


(SH(s)G(s)j 
18 


SEy(s) 


(2)  (2-33) 


(Sm(s)G(s))2  r(30%)SM(5) 

18  I  (2)(2.33) 


A 


J 
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Example  Calculation 


&  Assume  a  best  guess  size  at  SRR  of  50,000  ESLOC 
o  Assume  a  growth  factor  at  SRR  of  0.61 


^srr  \.M  o\ 


SI  - 

°SRR 


S M SRR  srr  +3) 


r(30%)S  s'2 


M  SRR 


V 


(2)(2-33) 


J 


c  - 

°SRR  — 


50,000(0.61  +  3)  |((50,000)(0.61))2  f  (30%)(50,000) 

18  +l  (2)(2.33) 


A 


•••Ssrr  =[60,167  7,877] 
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Probability  Density 
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Combined  PDF 


PDF 


Probability  Density  versus  Software  Size 


Software  Size  (effective  source  statements) 
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Confidence  Probability 


Seer 


Combined  CDF 


CDF 


Confidence  Probability  versus  Software  Size 


Software  Size  (effective  source  statements) 
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Review 


9  Measurement  objectifies  management 
9  Estimation  is  a  function  of  progress  (continuous  process) 

A  well-formed  estimate  is  specified  as  a  probability  distribution 
«  Uncertainty 

•  Variability 

•  Risk 

•  Opportunity 

»  Software  size  estimates  <■ 

•  Size  growth 

•  Size  estimation  variability 
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Software  Size  Growth  and  Uncertainty 


Both  Affect  Estimate  Quality  and  Project  Risk 

Mike  Ross 

Galorath  Incorporated 
100  North  Sepulveda  Boulevard 
Suite  1801 

El  Segundo,  California  90245 
(480)  488-8366  (phone)  (480)  488-8420  (fax) 
mross@qalorath.com  http://www.qalorath.com 


Abstract.  Examination  of  currently-accepted  software  cost,  schedule,  and 
defect  estimation  algorithms  reveals  a  common  acknowledgment  that 
estimated  software  size  is  the  single  most  influential  independent  variable. 
Unfortunately,  “The  most  important  business  decisions  about  a  software 
project  are  made  at  the  time  of  minimum  knowledge  and  maximum 
uncertainty.  ”  This  includes  minimum  knowledge  and  maximum 
uncertainty  about  a  software  product’s  effective  size  at  the  time  when 
most  estimating  is  done.  Further  complicating  the  issue  of  estimate 
quality,  in  the  author’s  opinion,  is  the  lack  of  a  commonly-accepted 
taxonomy.  This  paper  proposes  definitions  for  and  the  relationship 
between  two  key  attributes  of  software  size  estimates:  growth  and 
estimation  process  variability ,  both  being  distributions,  the  dispersions  of 
which  decrease  as  a  function  of  project  progress. 
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Introduction 


Purpose 

This  paper  proposes  definitions  for  and  the  relationship  between  two  key  attributes  of 
software  size  estimates:  growth  and  estimation  process  variability ,  both  being 
distributions,  the  dispersions  of  which  decrease  as  a  function  of  project  progress. 

Scope 

This  paper  focuses  on  handling  size  growth  and  variability  with  cost  and  schedule 
estimation  methods  that  employ  parametric  estimating  techniques;  however,  in  the 
author’s  opinion,  these  ideas  could  readily  be  extended  to  include  any  cost  and 
schedule  estimation  method.  The  issues,  assumptions,  and  propositions  presented  in 
this  paper  apply  to  all  software  development  projects  regardless  of  application  domain 
or  Software  Development  Life  Cycle  (SDLC)  paradigm. 

Background 

Examination  of  currently-accepted  software  cost,  schedule,  and  defect  estimation 
algorithms  reveals  a  common  acknowledgment  that  assumed  software  size  is  the  single 
most  influential  independent  variable.  It  follows  then  that  assumed  software  size  has  a 
significant  impact  on  a  given  estimate’s  quality  or  usefulness.  Unfortunately,  “The  most 
important  business  decisions  about  a  software  project  are  made  at  the  time  of 
minimum  knowledge  and  maximum  uncertainty.  ”[5]  This  includes  minimum  knowledge 
and  maximum  uncertainty  about  a  software  product’s  effective  size  at  the  time  when 
most  estimating  is  done  [5].  Further  complicating  the  issue  of  estimate  quality,  in  the 
author’s  opinion,  is  the  lack  of  a  commonly-accepted  taxonomy. 


Relevant  Taxonomy  and  Context 

Software  Development  Taxonomy 

Terms  defined:1 

■  Abstraction  -  A  representation  of  an  idea  or  concept  expressed  in  a  particular 
medium  or  language. 

■  Desire  -  A  want  or  need. 


1  Term  definitions  extracted  from  [5]. 
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■  [Software]  Requirements  -  An  abstraction  of  a  desire  for  which  computer 
technology  is  thought  to  be  a  viable  solution;  the  essence  of  a  software  product. 

■  Software  -  An  abstraction  of  a  desire  expressed  as  instructions  and  data  in  a 
form  that  can  be  acted  upon  by  a  computer. 

■  Process  -  A  set  of  actions  or  operations  conducing  to  an  end  [4]. 

■  Software  Development  Process  -  A  generalized  set  of  related  activities  that 
transform  desires  into  software. 

■  Software  [Development]  Project  -  A  specific  instance  of  a  software 
development  process. 

■  Software  Product  -The  primary  (deliverable)  result  of  a  Software  Development 
Project;  the  implementation  of  a  software  product. 

Software  Development  Process  Context 

Figure  1  depicts  the  context  of  a  software  development  process;  i.e.,  how  it  interfaces 
with  its  environment.  All  instances  of  software  development  processes  seek  to 
transform  software  requirements  into  a  software  product.  To  accomplish  this 
transformation,  they  consume  energy  in  the  form  of  labor  (people  doing  work)  from 
project  initiation  to  project  completion.  Since  no  software  development  process  is  a 
perfect  machine,  it  produces  some  amount  of  waste  or  entropy  (undesired  byproducts). 


Desire 


Laboi 


Technology 


Software 


Figure  1 :  Software  Development  Context2 


2  Figure  reprinted  with  permission  from  Michael  A.  ROSS  Consulting  &  Training.  All  rights  reserved. 
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Measuring  the  Software  Development  Process 

The  key  to  effectively  and  efficiently  measuring  the  software  development  process  is  to 
pick  measures  that  quantify  the  process’s  connections  to  its  surrounding  environment. 
Just  about  any  core  set  of  software  development  process  measures  will  include  the 
following: 

■  Size  -  An  abstraction’s  mass,  inertia,  bigness  (as  it  directly  relates  to  the  work 
that  must  be  done). 

■  Duration  -  The  elapsed  calendar  time  between  process  initiation  and  process 
completion. 

■  Effort  Cost,  Staffing  -  People  doing  work  during  the  software  development 
process  and  their  associated  cost,  over  elapsed  calendar  time. 

■  Quality  -  Defect  discovery  and  removal  over  elapsed  calendar  time. 


Effective  Technology 
(coefficient) 


„  Staffing  _ 
(people,  time) 

Effort 

(person-months) 

i 

Labor 

„  Cost  t 

(currency) 

Technology 


~ f 

Size 

(work  units) 

A 


Desire 


Start 


Finish 


Size 

(work  units) 


Time 

(calendar  months) 


Figure  2:  Software  Development  Process  Context  with 
Measurement3 


A  strong  indication  of  the  usefulness  of  these  measures  is  the  fact  that  they  address  the 
most  frequently  asked  questions  about  software  development  projects: 

■  How  big  will  the  product  be  when  delivered? 

■  How  long  is  it  going  to  take  ? 

■  How  many  people  will  be  needed  and  when? 


3  Figure  reprinted  with  permission  from  Michael  A.  ROSS  Consulting  &  Training.  All  rights  reserved. 


October  24,  2005 


Page  4  of  1 8 


■  How  much  will  it  cost? 

m  How  reliable  will  the  product  be  when  delivered? 


Attributes  of  Estimated  Software  Size 

Best  Guess  Size  Estimate  Defined 

People  generally  think  of  software  size  as  a  count  of  the  number  of  lines  of  code  or  the 
number  of  function  points  that  will  eventually  be  contained  by  a  to-be-developed 
software  product;  this  count  representing  some  sort  of  best  guess  SM  .  Natural  and 

relevant  questions  should  include,  “What  considerations  are  included,  what 
considerations  are  omitted,  how  confident  are  we  in  this  best  guess,  how  much 
uncertainty  surrounds  this  best  guess,  and  how  might  this  best  guess  change  over 
elapsed  calendar  time  during  the  software  development  project?” 

First  of  all,  since  we  are  concerned  about  how  this  best  guess  might  change  over 
elapsed  calendar  time  we  change  our  best  guess  representation  to  be  a  function  of 
progress  SM  (5)  where  progress  5  in  this  context  is  defined  to  be  normalized  earned 

value ;  i.e.,  the  project  starts  at  s  =  0%  complete  and  finishes  at  s  =  100%  complete  [6]. 

Second,  the  mention  above  of  confidence  and  uncertainty  and  the  use  of  the  term  best 
guess  implies  something  that  has  a  stochastic  nature;  i.e.,  there  exists  a  set  (a 
distribution)  of  numerous  possible  outcomes,  our  best  guess  being  but  one  element  of 
this  distribution.  We  therefore  postulate  that  a  well-formed  estimate  is  specified  in  terms 
of  a  selected  probability  distribution  and  its  attributes.  It  seems  reasonable  then  to 
assume  that  our  best  guess  represents  some  sort  of  central  tendency  of  its  associated 
distribution.  It  also  seems  reasonable  to  assume  that  this  distribution  is  continuous 
rather  than  discrete.4  The  next  set  of  questions  are,  “What  kind  of  distribution  are  we 
talking  about  (Uniform,  Normal,  Beta,  Triangular,  etc.) ,  what  are  its  attributes 
( location ,  dispersion,  number  of  modes,  skewness,  kurtosis,  etc.),  and  which  form  of 
central  tendency  does  this  best  guess  represent  (mean,  median,  mode,  other)?” 


Distribution  Functions 

There  has  been  and  continues  to  be  much  debate  over  which  distribution  function  best 
represents  estimated  software  size  uncertainty.  The  leading  candidates  for  this  honor 
are  (in  no  particular  order):5 


4  The  author  acknowledges  that  software  size,  being  a  count  of  something,  could  be  viewed  as  discrete  rather  than  as  continuous;  however, 
since  the  range  of  possible  outcomes  is  relatively  large  and  the  resolution  of  possible  outcomes  is  relatively  fine,  the  author  chooses  to  view 
the  distribution  as  continuous. 

5  Equations  for  these  distributions  and  their  attributes  can  be  found  at  http://www.mathworld.wolfram.com. 
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■  Normal  (Gaussian)  Distribution 

■  Bi-Normal  Distribution6 

■  Triangular  Distribution 

■  Beta  (special  case  of  a  Weibull)  Distribution 


Location  (Central  Tendency) 

We  have  already  suggested  that  a  best  guess  represents  some  sort  of  central  tendency 
of  its  associated  uncertainty  distribution.  Intuitively,  of  the  three  most  common 
measures  of  central  tendency  (mean,  median,  and  mode),  it  is  the  mode  that  seems  to 
best  represent  the  idea  of  a  best  guess.  For  example,  I  might  say  something  like,  “If  I 
were  to  run  this  project  many  times  (approaching  infinity),  I  believe,  based  on  what  I 
know  today,  that  a  final  size  outcome  of  about  50,000  effective  source  statements  would 
happen  more  times  than  any  other  final  size  outcome.  In  other  words,  approximately 
50,000  effective  source  statements  is  thought  to  be  the  most  likely  or  mode  value  of  our 
size  uncertainty  distribution.  It  follows  then  that  best  guess  and  most  likely  are 
synonymous  within  the  context  of  this  discussion.  Mathematically,  this  value  is  the 
global  maximum  of  our  size  uncertainty  distribution’s  Probability  Density  Function 
(PDF). 


Dispersion 

Of  the  three  most  common  measures  of  dispersion;  mean  deviation,  interquartile  range, 
and  standard  deviation  o ;  the  latter  is  almost  invariably  used  by  statisticians  [2],  It  is 
defined  as  the  square  root  of  the  second  central  moment  m2  (variance)  which,  in  turn,  is 

defined  as  the  average  of  the  squared  deviations  from  the  mean. 


Modes,  Skewness  and  Kurtosis 

We  make  the  simplifying  assumption  that  the  distributions  representing  element-level 
contributors  to  uncertainty  are  unimodal;  however,  combinations  of  multiple  distributions 
can  yield  multimodal  distributions. 

Skewness  (asymmetry  to  the  left  or  right)  and  kurtosis  (peakedness)  are  measured  as 
functions  of  the  third  and  fourth  central  moments  m3  and  m4  respectively.  Skewness  is 

presented  here  since  most  estimators  agree  that  size  uncertainty  distributions  are 
asymmetric  and  tend  to  be  right  skewed  (long  tail  on  the  right  side  of  the  PDF).  Kurtosis 
does  not  seem  to  be  too  much  of  an  issue  at  this  point;  however,  as  more  data  is 


6  Combines  the  left  half  of  one  Normal  Distribution’s  PDF  having  a  standard  deviation  C7Low  with  the  right  half  of  another  Normal 
Distribution’s  PDF  having  a  standard  deviation  <JHigh  in  order  to  model  skewness  using  Normal  Distribution  math. 
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collected  that  can  be  used  to  relate  size  estimates  with  size  outcomes,  it  may  become 
more  relevant  to  describing  the  ideal  size  uncertainty  distribution. 


Uncertainty  Defined 

We  now  introduce  an  emerging  model  that  defines  the  notion  of  uncertainty  as  a 
function  of  variability,  risk,  and  opportunity  [3]  and  use  the  following  conceptual  model 
as  a  guide  for  describing  size  uncertainty7. 

U  =LV  +  (LR-LO)  Eqn. ' 


where: 

U  Uncertainty:  a  random  variable  representing  the  uncertainty  about  a 

particular  value  or  metric,  expressed  as  a  probability  distribution  of 
possible  outcomes. 

V  Variability:  a  random  variable  representing  the  impact  on  the 

particular  value  or  metric  by  an  event  or  events  that  will  occur 
(probability  of  1),  expressed  as  a  probability  distribution  of  possible 
outcomes. 

R  Risk:  a  random  variable  representing  the  impact  on  the  particular 

value  or  metric  by  a  specific  unfavorable  event  that  may  or  may  not 
occur  (there  exists  some  known  probability  of  occurrence). 

O  Opportunity:  a  random  variable  representing  the  impact  on  the 

particular  value  or  metric  by  a  specific  favorable  event  that  may  or 
may  not  occur  (there  exists  some  known  probability  of  occurrence). 


Two  Key  Drivers  of  Software  Size  Estimates 

Within  the  software  estimation  community  and  its  serviced  stakeholder  organizations, 
there  has,  for  better  or  worse,  evolved  two  sometimes  complementary  and  sometimes 
conflicting  terms:  size  growth  and  size  uncertainty.  We  propose  the  following  definitions 
for  these  two  terms  in  the  hope  that  some  of  the  inherent  conflict  can  be  understood  and 
minimized. 

■  Size  Growth  -  Variability  in  the  baseline  estimated  software  size  that  results 
from  a  change  in  the  common  understanding  of  the  required  functionality  and/or 
the  context  in  which  the  software  development  project  and  its  resultant  software 
product  exist.  Note  that  we  do  not  characterize  size  growth  as  a  risk  in  our 


7  Use  of  the  summation  symbols  in  the  conceptual  relationship  is  intended  to  show  aggregation  of  multiple  contributing  random  variables, 
each  of  which  may  be  multivariate  in  nature.  The  summation  symbols  do  not  imply  that  simple  arithmetic  addition  is  appropriate;  it  is  more 
likely  that  simulation  techniques  such  as  Monte  Carlo  will  be  required  to  properly  evaluate  the  contributors  to  uncertainty. 
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model.  We  assume  that  size  growth  will  occur  and  that  it  embodies  the  impact  of 
those  events  not  yet  known  and  specified  (not  yet  characterized  as 
risks/opportunities).  Note  also  that  we  have  narrowed  the  focus  of  the  term  size 
growth  to  one  of  a  technological  and  programmatic  nature.  This  definition  implies 
the  desirability  to  find  some  sort  of  growth  factor  function  that  can  predict 
additional  size  and  its  associated  uncertainty.  As  the  project  matures,  we  expect 
that  risks  and  opportunities  will  become  known  and  specified  and,  therefore, 
removed  from  consideration  as  a  part  of  variability.  This  implies  a  desire  to 
express  growth  factor  as  a  function  of  progress  on  a  given  project. 

■  Size  (Estimation)  Uncertainty  -  Variability  that  results  from  the  stochastic 
nature  of  human  behavior  and  model  behavior  associated  with  the  software  size 
estimation  process.  Note  that  we  have  narrowed  the  focus  of  the  term  size 
uncertainty  (hereinafter  referred  to  as  size  estimation  variability)  to  one  of 
process  rather  than  to  one  that  could  be  assumed  to  encompass  all  estimated 
size  uncertainty;  i.e.,  size  growth  and  size  estimation  variability  are  mutually 
exclusive.  Size  estimation  variability  is  described  by  a  specific  distribution 
(including  its  attributes)  of  possible  software  size  impacts  given  some  common 
understanding  of  the  required  functionality  and  of  the  context  in  which  the 
software  development  project  and  its  resultant  software  product  exist. 


Size  Growth 

Software  project  management  would  be  a  whole  lot  simpler  if  we  knew,  from  the 
beginning,  precisely  how  big  the  software  will  end  up  being.  Issues  of  efficiency  would 
then  be  the  sole  source  of  cost  and  schedule  uncertainty.  Unfortunately,  static  software 
size  is  not  reality.  A  rare  project  experiences  no  requirements  changes  and  no 
expansion  of  scope.  This  is  a  serious  issue  since  variations  in  software  size  have  the 
single  largest  influence  on  software  development  time,  effort,  cost,  staffing,  and  the 
number  of  delivered  defects  [5]. 

We  have  previously  stated  that  size  growth  stems  from  context  volatility.  In  order  to 
understand  these  notions  of  size  growth  and  context  volatility  we  must  understand  its 
source.  If  one  were  to  solicit  a  list  of  things  that  cause  software  size  to  grow  it  might 
include  some  of  the  following: 

■  The  customer  doesn’t  know  what  he/she  wants. 

■  The  customer  doesn’t  understand  the  problem. 

■  The  mission  has  changed. 

■  The  regulations  that  govern  how  this  software  should  behave  have  changed. 

■  The  vendor  added  a  few  extra  features  that  he/she  thought  the  customer  would 
like. 
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■  The  vendor  finished  early  so  the  customer  and/or  the  vendor  thought  up  a  few 
things  to  add. 

Analysis  of  the  preceding  list  suggests  the  following  possible  organization  of  issues  that 
influence  software  size  growth: 

■  Operational  Environment  Volatility 

■  Essence  (Requirements)  Volatility 

■  Essence  Understanding  (Requirements  Completeness  and  Correctness) 

■  Essence  versus  Implementation  Correspondence 

All  of  the  preceding  issues  seem  to  fall  into  either  the  Technical  or  the  Programmatic 
Risk8  Driver  categories  per  [1]. 

We  earlier  suggested  the  desire  for  a  growth  factor  function  that  can  predict  additional 
size  as  a  function  of  progress  on  a  given  project.  Analysis  of  historical  data  collected  by 
Galorath  Incorporated  suggests  that  this  growth  factor  function  G(s)  is  linear  and  is 

approximately: 

G  (s)  —  -0.1s  +  0.69  Eqn.  2 


For  example,  if  we  assume  that  a  project’s  normalized  earned  value  when  Software 
Requirements  Analysis  is  complete  (at  Software  Requirements  Review  or  SRR)  to  be 
1 1 .8%,  then9: 

GSrr  =  G(1 1 .8%)  =  -0.7  (1 1.8%)  +  0.69  =  0.61  Eqn. 


Since  we  have  already  judged  the  issues  impacting  size  growth  to  be  technical  or 
programmatic  in  nature,  we  assume  that  our  size  growth  factor  distribution  can  best  be 
represented  as  a  Triangular  Distribution  per  [1]  described  by  the  parameter  vector 
G  (s): 


G(s)  =  [L  M  «]  =  [0  0  G(s)]  Eqn.  4 

where  L  is  the  lowest  conceivable  growth  factor  value,  M  is  the  most  likely  (mode) 
growth  factor  value,  and  H  is  the  highest  conceivable  growth  factor  value  [7]10.  If  our 


8  Note  that  this  usage  of  the  term  Risk  is  not  consistent  with  our  uncertainty  model  but,  rather,  refers  to  terminology  used  in  the  cited 
document. 

9  Note  that  this  example  is  consistent  with  the  results  documented  in  [7]. 

10  The  results  of  [7]  suggest  0  —  L  <M  <  H  .  To  preserve  consistency  with  our  system  of  distributions  we  have  transformed  the  results 
in  [7]  to  force  M  —  L  —  0  while  maintaining  the  total  area  under  the  Triangular  Distribution  PDF. 
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best  guess  of  software  size  at  SRR  SM  SRR  =  50,000  effective  source  statements  (based 

on  the  current  common  understanding  when  Software  Requirements  Analysis  is 
complete  of  the  required  functionality  and  of  the  current  common  understanding  of  the 
technology  to  be  applied),  then  the  size  growth  implication  distribution  is  a  Triangular 
Distribution  described  by  the  parameter  vector  SG  (s) : 

SgW  =  S„WG(S)=[0  0  S„(S)G(S)]  Eqn.  5 

SG_SRE  =  [o  0  SMjBR  (0.61)]  =[0  0  30,500] 


The  Probability  Density  Function  (PDF)  for  a  Triangular  Distribution  is  given  by: 


Pt™ - 1~,  (  x)  ^ 


L  Triangular  ' 


2(x-L)2 


1- 


(H-L)(M-L) 

2(H-xf 

(H-L)(H-M) 


for  xe  [ L,M  ] 
for  xe  [M,H] 


Eqn.  6 


The  Cumulative  Distribution  Function  (CDF)  for  a  Triangular  Distribution  is  given  by: 


D 


Triangular 


(x): 


(x-L)2 


1- 


(H-L)(M-L) 

{H-xf 

( H-L)(H-M ) 


for  xe  [L,M] 
for  xe  [M,H] 


Eqn.  7 


The  Probability  Density  Function  (PDF)  and  the  Cumulative  Distribution  Function  (CDF) 
for  the  Triangular  Distribution  described  by  SG  (5)  are  graphed  below  in  Figure  3  and 

Figure  4  respectively. 
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Confidence  Probability  Probability  Density 


PDF 

Probability  Density  versus  Software  Size 


Figure  3:  PDF  of  a  Triangular  Distribution  Described  by  SG  (5) 


CDF 

Confidence  Probability  versus  Software  Size 


Figure  4:  CDF  of  a  Triangular  Distribution  Described  by  SG  (s) 
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Size  Estimation  Variability 

Because,  until  project  completion,  software  size  must  be  estimated,  it  follows  that 
software  size  is  uncertain,  regardless  of  whether  or  not  we  recognize  size  growth.  The 
very  nature  of  the  word  estimate  implies  uncertainty.  We  assume  uncertainty  in  this 
context  to  mean  that  there  exists  some  distribution  (with  specific  attributes)  of  possible 
software  size  outcomes.  Therefore,  in  order  to  quantify  size  estimation  variability  we 
must  define  this  distribution  and  its  attributes.  Size  estimation  process  and  model 
variability  are  best  represented  by  a  Normal  (Gaussian)  Distribution  [1],  We  assume, 
based  on  [7],  a  (±30%)SM  (5)  conceivable  range  of  this  distribution;  conceivable  being 

defined  as  ±2.33<r  (between  the  1st  to  the  99th  percentiles). 

Continuing  our  example  situation,  if  our  best  guess  of  software  size  at  SRR  is 
SM  Srr  =50,000  effective  source  statements  (based  on  the  current  common 

understanding  when  Software  Requirements  Analysis  is  complete  of  the  required 
functionality  and  of  the  current  common  understanding  of  the  technology  to  be  applied), 
then  the  amount  of  additional  estimated  software  size  due  to  size  estimation  variability 
is  a  Normal  Distribution  described  by  the  parameter  vector  SEV  (5) : 


^EV  ( 'S  )  [A  G] 


(30%)Sm  (£) 
(2)(2.33) 


S 


EV 


0 


(30%)  ^ 
(2)(2.33) 


=  [0  3,219] 


Eqn.  8 


where  //  is  the  arithmetic  mean  of  the  distribution  (in  effective  source  statements)  and 

a  is  the  standard  deviation  of  the  distribution  (also  in  effective  source  statements).  Note 
that  we  specify  //  to  be  0  in  order  to  center  the  distribution  about  0 ,  the  left  half 

(negative)  representing  size  decrease  due  to  estimation  variability,  the  right  half 
(positive)  representing  size  increase  due  to  estimation  variability. 

The  Probability  Density  Function  (PDF)  for  a  Normal  (Gaussian)  distribution  is  given  by: 

/  3  1  ,  3  Eqil.9 

PNormalW  = - 7T=  e  ^  for  X  £  (-00,  00) 

o^2n 


There  exists  no  closed  form  representation  of  the  Cumulative  Distribution  Function 
(CDF)  for  a  Normal  (Gaussian)  Distribution;  however,  Microsoft®  Excel  contains  a  built- 
in  approximation  function  for  this  purpose.  Additionally,  a  reasonable  second  order 
polynomial  approximation  is  given  by: 
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D  Normal  (-*0 


0.0903 


-0.0903 


^  x-ju^ 

2 

+  0.4207 

X  —  jU^ 

{  0  ) 

l  O’  J 

^  X-JU^ 

2 

+  0.4207 

r  x- tO 

{  <7  J 

l  <7  ) 

0.01  for  xe  (-oo,^-2.33<j]  Eqn.  10 
+  0.5  forxe  [//-2.33<r,//) 

0.5  for  x  -  fi 

+  0.5  forxe  (ju,ju  +  2.33(y\ 

0.99  forxe  [//  +  2.33cr,°o) 


The  Probability  Density  Function  (PDF)  and  the  Cumulative  Distribution  Function  (CDF) 
for  the  Normal  Distribution  described  by  SEV  (5)  are  graphed  below  in  Figure  5  and 

Figure  6  respectively. 


PDF 

Probability  Density  versus  Software  Size 


Software  Size  (effective  source  statements) 

Figure  5:  PDF  of  a  Normal  Distribution  Described  by  Sy  (5) 
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CDF 

Confidence  Probability  versus  Software  Size 


Software  Size  (effective  source  statements) 


Figure  6:  CDF  of  a  Normal  Distribution  Described  by  Sv  (5) 


Combining  Size  Growth  and  Size  Estimation  Variability 

Based  on  our  definitions  of  size  growth  and  of  size  estimation  variability,  we  can  sum 
our  best  guess  size  estimate  SM  (5) ,  our  size  growth,  a  Triangular  Distribution  described 

by  SG  (.v) ,  and  our  size  estimation  variability,  a  Normal  Distribution  described  by  SEV  (5) ; 

the  result  being  our  estimated  size  distribution  of  unknown  type  and  described  by  the 
parameter  vector  S(s)  =  [//  cr],  /i  being  its  arithmetic  mean  and  o  being  its  standard 
deviation.  In  order  to  solve  for  n  and  a  we  can  take  advantage  of  two  statistical 
theorems  as  described  in  [2]  and  [1],  one  for  expectation  E  that  yields  fi  and  one  for 

variance  V  that  yields  cr2 .  Each  can  be  applied  to  a  series  of  independently11 * 
distributed  random  variables  X  : 


and 


(  n 


1*. 


V  i= 1 


!*(*.) 


V 


n  n 

Zx<  =ZV(X.) 


<= 1 


i=l 


Eqn.11 


Eqn.  12 


11  Independence  is  a  necessary  prerequisite  for  the  variance  theorem;  however,  it  is  not  a  necessary  prerequisite  for  the  expectation  (mean) 

theorem. 
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Eqn.  13 


Given  our  series  of  independent  random  variables  SM  (5) ,  SG  (5) ,  SEV  (.v) : 

Rs M(s)  is) 

_0  +  0  +  Sm(J)G(J)_Sm(J)G(J) 

2  3 

^SEV(s)  =  0 

.  u  =s  L"f  I  Sm  -  Sm  (-HG(-)+3) 

*  ■  ^sfs)  r/  ^  ^ 

and 


^SmU)  0 


®g(s) 


SEv(5) 


L2  +M2  +  H2 -LH-LM -MH  _ 

18 

(30%)S„  (s) 

(2)  (2.33) 

|(s„MGM)2  | 

f(30%)S„(s)Y 

18 


18 


(2)  (2.33) 


Eqn.  14 


Continuing  our  example  size  estimate  taken  at  SRR  where  our  best  guess 
Sm_Srr  =  50,000  and  our  size  growth  factor  GSRR  =  0.61 : 


C  _ 
°SRR 


$MJSRR  (  G SRR  +  3  )  I  (  $M_SRR  & SRR  ) 


18 


+ 


r(30%)S  V 


JM_SRR 


(2) (2.33) 


SRR 


50,000(0.61  +  3)  J((50,000)(0.61))2  ( (30%)  (50, 000)' 


18 


(2)(2.33) 


.-.SSRR  =[60,167  7,877] 


Eqn.  15 


We  now  know  the  mean  or  expected  value  (60,167  effective  source  statements)  and 
standard  deviation  (7,877  effective  source  statements)  of  the  statistical  sum  of  the  three 
contributors  to  our  size  estimate.  If  this  were  one  small  component  in  a  larger  whole 
consisting  of  many  components,  then  we  could  take  advantage  of  the  Central  Limit 
Theorem  “which  states  that  the  sum  of  a  large  number  of  independent  random 
variables  will  be  approximately  normally  distributed  almost  regardless  of  their 
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individual  distributions.  ”[2]  Unfortunately,  we  don’t  have  a  large  number  of 
independent  random  variables  in  this  example;  thus,  if  we  wish  to  extract  probability  / 
confidence  information  from  our  size  estimate  distribution,  we  are  left  with  a  problem 
that  is  best  solved  by  a  calculator  or  a  software  product  that  uses  simulation. 


Summary  and  Conclusion 

Purpose  Revisited 

This  paper  proposed  definitions  for  and  the  relationship  between  two  key  attributes  of 
software  size  estimates:  growth  and  estimation  process  variability,  both  being 
distributions,  the  dispersions  of  which  decrease  as  a  function  of  project  progress. 

Areas  for  Further  Study 

The  following  are  suggestions  for  furthering  the  discussion  of  software  size  growth  and 
uncertainty: 

■  Collect  more  (and  more  continuous)  size  estimation  data  and  use  it  to  strengthen 
size  growth  factor  functions. 

■  Investigate  making  the  conceivable  range  of  the  size  estimation  variability 
distribution  be  a  function  of  project  progress  (i.e.,  factor  in  the  notion  of  project 
maturity  and  associated  learning). 

■  Investigate  relevant  methods  and  techniques,  avoiding  simulation,  that  provide 
probability  /  confidence  information  from  distributions  that  are  the  statistical  sum 
of  a  small  number  of  constituent  distributions  (i.e.,  the  resulting  distribution  is 
unlikely  to  be  a  Normal  Distribution). 


References 

[1]  Book,  S.A.,  “Cost-Risk  Computations  by  Hand  Calculator”;  Proc.  SCEA  National 
Conference  &  Educational  Workshop,  The  Society  of  Cost  Estimating  and 
Analysis,  Scottsdale,  AZ,  June  2002. 

[2]  Bulmer,  M.G.,  Principles  of  Statistics,  Dover  Publications,  Inc.,  New  York,  NY, 
1979. 

[3]  Hutchings,  Christopher,  “Risk  is  not  a  four  letter  word!”,  Proposed  Seminar: 
Galorath  Inc.,  El  Segundo,  CA,  2005. 

[4]  Mish,  F.  (Editor  in  Chief),  Merriam-Webster’ s  Collegiate  Dictionary,  Tenth 
Edition,  Merriam-Webster,  Incorporated,  Springfield,  MA,  1999. 


October  24,  2005 


Page  1 6  of  1 8 


[5]  Ross,  M.,  “Managing  Software  Size,”  Proc.  Joint  ISPA  / SCEA  2003  Conference, 
The  International  Society  of  Parametric  Analysts  and  The  Society  of  Cost 
Estimating  and  Analysis,  Orlando,  FL,  June  2003. 

[6]  Ross,  M.,  “Parametric  Project  Monitoring  and  Control,”  Proc.  Joint  ISPA/  SCEA 
2005  Conference,  The  International  Society  of  Parametric  Analysts  and  The 
Society  of  Cost  Estimating  and  Analysis,  Denver,  CO,  June  2005. 

[7]  Tarbet,  D.,  “Software  Cost/Schedule  Estimation:  Code  Growth,”  Internal  White 
Paper:  Galorath  Inc.,  El  Segundo,  CA,  2002. 


October  24,  2005 


Page  1 7  of  1 8 


Biography 

Michael  A.  Ross  has  over  30  years  of  practical  experience  in  software  engineering  as  a 
developer,  manager,  process  champion,  consultant,  instructor,  and  award-winning 
international  speaker. 

Mr.  Ross  is  currently  the  Chief  Engineer  of  Galorath  Incorporated,  makers  of  the  SEER 
suite  of  estimation  tools,  where,  for  the  past  three  years,  he  has  been  responsible  for 
the  advancement  and  realization  of  the  technology  aspects  of  Galorath’s  mission  and 
vision. 

Prior  to  joining  Galorath,  Mr.  Ross  was  Vice  President  of  Education  Services  for 
Quantitative  Software  Management,  Inc.  (makers  of  the  SLIM  suite  of  software 
estimating  tools).  He  was  responsible  for  the  development  and  delivery  of  all  QSM 
training.  During  his  seven-year  tenure  with  QSM,  he  served  as  one  of  the  company’s 
primary  consultants  and  analysts  working  with  Fortune  500  companies  and  government 
agencies  in  the  areas  of  software  measurement,  sizing,  estimating,  tracking, 
forecasting,  and  benchmarking. 

Mr.  Ross,  during  17  years  with  Honeywell  Air  Transport  Systems  (formerly  Sperry  Flight 
Systems)  and  2  years  with  Tracor  Aerospace,  developed  or  managed  the  development 
of  embedded  software  for  avionics  systems  installed  various  commercial  airplanes 
including  the  Boeing  737-500,  757,  767,  777,  the  Douglas  MD-11,  the  Lockheed  LI  01 1  - 
500,  the  British  Aerospace  BAe-146,  the  Airbus  A320;  and  for  expendable 
countermeasures  systems  installed  in  various  military  aircraft  and  missiles.  He  also  co¬ 
founded  Honeywell  Air  Transport  Systems’  process  improvement  team  (later  to  become 
its  SEPG),  served  as  its  focal  for  software  project  management  process  improvement, 
and  served  as  a  Honeywell  corporate  SEI  CMM  assessor. 

Mr.  Ross  did  his  undergraduate  work  at  the  United  States  Air  Force  Academy  and 
Arizona  State  University,  receiving  a  Bachelor  of  Science  in  Computer  Engineering.  He 
is  a  member  of  the  Project  Management  Institute  (PMI),  the  Institute  of  Electrical  and 
Electronics  Engineers  (IEEE),  the  International  Function  Points  Users  Group  (IFPUG), 
the  International  Society  of  Parametric  Analysts  (ISPA),  the  Society  of  Cost  Estimating 
and  Analysis  (SCEA),  the  Arizona  Software  Association,  and  the  Phoenix  area  Software 
Process  Improvement  Network. 
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Summary 


9  Software  projects  fail  more  often  than  not 
9  Project  success  Good  management 
9  Measurement  objectifies  management 
o>  Software  projects  are  governed  by  dynamic  properties 

•  Properties  currently  accounted  for  in  the  Project  Planning  process 

•  Properties  should  also  be  accounted  for  in  the  Project 
Monitoring  and  Control  process 

o>  Project  Monitoring  Performance  Measurement 
o>  4-D  Earned  Value  objectifies  progress 
»  Project  Control  Control  Limits 
»  Re-Baselining  Performance-Based  Forecasting 

9  Communication  is  essential  to  successful  project  management 
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Things  are  Getting  Better;  however, 
There’s  Still  Room  For  Improvement 


Success:  The  project  is 
completed  on  time  and  on 
budget,  with  all  features 
and  functions 

Challenge:  Over  budget, 
over  time,  offers  fewer 
features  than  originally 
specified 

Failure:  Project  is 
cancelled  prior  to 
completion 


1994  1996  1998  2001 


How  does  ineffective 
management  of  resources 
(people,  time,  $)  contribute  to 
this  problem? 


2004 


The  Price  of  Failure 


□  Overruns 
■  Failure 


Source:  Standish  Group  International,  Inc. 
“CHAOS”  studieswww.standishgroup.com 
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Mnemonic  Aid  for 

Software  Project  Management 


9*  Planning  -  estimating,  scheduling 

9*  Resourcing  -  interviewing,  hiring,  motivating 

9  Organizing  -  establishing  interpersonal  communication  paths 
and  rules,  mapping  resources  to  tasks 

9  T raining  -  teaching,  mentoring 

9  equipping  -  acquiring  and  allocating  equipment,  tools, 
materials,  supplies,  products  etc. 

9  Controlling  -  directing,  measuring,  correcting  and/or  replanning 
9  transitioning  -  delivering,  reviewing,  analyzing,  archiving 
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Project  Management  Context 
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Process  Focus  (CMMI™) 


o  Project  Planning 

•  Establish  Estimates 

•  Develop  a  Project  Plan 

•  Obtain  Commitment  to  the  Plan 

9  Project  Monitoring  and  Control 

•  Monitor  Project  Against  Plan 

•  Manage  Corrective  Action  to 
Closure 

9  Measurement  and  Analysis 

•  Align  Measurement  and  Analysis 
Activities 

•  Provide  Measurement  Results 
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Software  Development  and 
Measurement 


^  Staffing  ^ 
(people,  time) 

Effort 

(person-months) 

T 

Labor 

Cost 

(currency) 

Size 

(work  units) 

_ 


Desire 


Start 1 


Effective  Technology 

(coefficient) 


Technology 


Friction 


Software 
Development 
Process 


Defects 

(count) 


Software 


Time 

(calendar  months) 


\ Finish 


Size 

(work  units) 
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Fundamental  Measures 


Size 

Effective  Technology 
Time 

Effort  ^  Cost ,  Staffing 
Defects 
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Estimate  Defined 


es*ti*mate  (es'ti  mit),  n. 

an  approximate  judgment  or  calculation ,  as  of  the 
value  or  amount  of  something 

a  prediction  that  is  equally  likely  to  be  above  or  below 

the  actual  result  (Tom  DeMarco) 

A  WELL  FORMED  ESTIMATE 
IS  A  DISTRIBUTION 


Metric  Metric 
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3  Laws  of  Software  Development 
_ Dynamics _ 


o  Brooks’  Law  (Software  Equation) 

•  Adding  people  to  a  late  project  makes  it  later. 

•  Development  time  (duration)  and  development  effort  (labor)  are  not  linearly 
interchangeable. 

9  Paul  Masson’s  Law  Applied  to  Software  Development  (Minimum  Time 
Equation) 

•  No  [software]  before  its  time . 

•  Each  and  every  project,  by  its  nature  (technical  difficulty),  can  effectively 
handle  only  so  much  staffing  acceleration;  therefore,  there  exists,  for  each 
and  every  project,  some  minimum  achievable  development  time. 

o  Parkinson’s  Law  (Optimal  Effort  Equation) 

•  Work  expands  so  as  to  fill  the  time  available  for  its  completion. 

•  There  exists,  for  each  and  every  project,  some  point  of  maximum 
productivity;  i.e.,  some  point  that  represents  the  most  efficient  use  of  labor 
on  the  project. 
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Software  Development  Dynamics 


For  a  given  Size  and  Technology 


Elapsed  Calendar  Time  (months) 
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Software  Project  Management 
Out  of  Control  Process 
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Software  Project  Management 
Ad  Hoc  Process 
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Software  Project  Management 
Partially  Managed  Process 
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Software  Project  Management 
Fully  Managed  Project 


©  Galorath  Inc.  2005 
All  Rights  Reserved 


October  24,  2005 


15 


Seer 


Performance  Measurement: 
Measures  and  Metrics 


9  Fundamental  Cost  of  Work  Measures 

•  Baseline  Budget  -  Budget  at  Completion  (BAC) 

•  Planned  -  Budgeted  Cost  of  the  Work  Scheduled  (BCWS) 

•  Earned  -  Budgeted  Cost  of  the  Work  Performed  (BCWP) 

•  Spent  -  Actual  Cost  of  the  Work  Performed  (ACWP) 

o  Variances  (Differences  between  Cost  of  Work  Measures) 

•  Schedule  Variance  (SV) 

•  Cost  Variance  (CV) 

•  Budget  Variance  (BV) 

•  Time  Variance  (TV) 
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Performance  Measurement: 
Measures  and  Metrics 


9  Performance  Indices  -  (Ratios  Between  Cost  of  Work  Measures) 

•  Schedule  Performance  Index  (SPI) 

•  Cost  Performance  Index  (CPI) 

•  Budget  Performance  Index  (BPI) 

•  Time  Performance  Index  (TPI) 

•  Composite  Performance  Index  (XPI) 

•  To-Complete  Performance  Index  (TCPI) 

•»  Status  and  Forecasting  Metrics 

•  Estimate  at  Completion  (EAC) 

•  Estimate  to  Complete  (ETC) 
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Three  Unit  Systems  for 
Performance  Measurement  Values 


9  Monitary  Value  -  units  of  currency;  e.g.: 

•  $ 

•  £ 

•  € 

9  Effort  Value  -  units  of  labor;  e.g.: 

•  person-hours,  staff-hours,  effort-hours,  labor-hours 

•  person-months,  staff-months,  effort-months,  labor-months 

»  Normalized  Value  -  unitless 

•  %  of  full  scale 
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4-D  Earned  Value 


9  SDLC  Primary  Activity  Completion 
»  Artifact  Completion 
«  Milestone  Completion 
9>  Defect  Discovery  /  Removal 
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Schedule  Accomplishments  Chart 


il  PPMC  Schedule  Accomplishments 


s® 


%  of  Baseline  Ran 
192- 


CDR  -  Snapshot 


BCWP 

ACWP 

BOWS 

Forecasted  Date: 

11-07  ^ 

Forecasting  Method: 

Performance 

rid 

BV/BPI  | 

k  i  V 

1 

SV/SPI 1 

r  i  r 

BOWS 

1.1 

|CY/CP/| 


1-05  3-05  5-05  7-05  9-05  11-05  1-06  3-06  5-06  7-06  9-06  11-06  1-07  3-07  5-07  7-07  9-07  11-07  Months 

Gasoline  fit  Complete:  55777  Hours;  34.44  Months 
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Example  Project:  Metrics  Charts 

at  Project  Start  (Initial  Plan) 
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Example  Project:  Metrics  Charts 

at  System  Design  Review 
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Example  Project:  Metrics  Charts 

at  Software  Requirements  Review 


IS  Status  Dashboard  -  Program:  CPU  1 


El® 


ppmc  sch  ed  li  I  e  Acco  m  pi  ish  me  nts 


%  of  Base  re  Pla  n 
434J0-,- 


CPU  1 


Currtrir  Elr. 

Eiirltd  Vi  I  Lit 
A™al  Sptni 
Bflatlirit  H«n 

FfcttMifc  End  Dirt:  4/2S/OS 

Ft-rttiiidhl  Mt-iltd:  F'trtwT'Mritt 

UirUjidirt:  S&BtfS  TT 


1-GS  AOS  7-OS  IOCS  1-06  4-0*6  7-06  1006  1-07  A07  7-07  1007  1-0® 
IzBefll  GompLeiC;  142557  House  36.35  XtoTtlto 


Defects  Tracking 


Performance  Indices 


Sched  ule 

Variance 

Time 

Variant 

Cost 

Variance 

Size 

Growth 

Defects 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 
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Example  Project:  Metrics  Charts 

at  Preliminary  Design  Review 


HI,  Status  dashboard  -  Program:  CPU  1 

BE® 

PPMC  Schedule  Accomplishments 

Performance  Indices 

n 

%  of  Base  ne  Ran 


CPU  1 


EjsAeUCamfzl£73J4€  Hfcwrs:  3AJLQ 


Indes 

3.00 


CPU  1  (Cumulative) 


0  cpi 


ASR1 


jlI 


1-05  3-05  5-05  7-05  9-05  11-05  1-OS  3-OS  Date 


Defects  Tracking 


Health  Status  Indicator 


Defects 
■400  i 

CPU  1 

Schedule 

Time 

Cost 

Size 

Variance 

Variance 

Variance 

Growth 

Defects 

320  ■ 

CPU  1 

BETTER 

BETTER 

BETTER 

WORSE 

WORSE 

240  ■ 

N/A 

N/A 

N/A 

N/A 

N/A 

If  ■ 

Estimated  Defects  Reported 

ISO  - 

If  □ 

Estimated  Defects  Removed 

f  1=1 

Actual  Defects  Reported 

N/-A 

N/A 

N/A 

N/A 

N/A 

/*  ■ 

Actual  Defects  Removed 

SO  - 

N/A 

N/A 

N/A 

N/A 

N/A 

1-05  . SOS 
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Example  Project:  Metrics  Charts 

at  Critical  Design  Review 


HI,  Status  dashboard  -  Program:  CPU  1 

BE® 

PPMC  Schedule  Accomplishments 

Performance  Indices 

n 

%  of  Bsse  re  Plain 


CPU  1 


E  as  t  T4  A"  5*777  HtoSMS  :  34.44  WonShs 


Defects  Tracking 


CPU  1 


Indes 

3..O0 


CPU  1  (Cumulative) 


0  CF1 


ASR1 


jlI 


1-05  3-05  5-05  7-05  9-05  11-05  1-OS  3-OS  5-OS  7-OS  Oat* 


Health  Status  Indicator 


Schedule 

Variance 

Time 

Variance 

Cost 

Variance 

Size 

Growth 

Defects 

BETTER 

WORSE 

BETTER 

NO  CHANGE 

WORSE 

N/A 

N/A 

N/A 

N/A 

N/A 

N/-A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 
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Performance-Based  Forecasting  and 
_ Re-Baselining _ 


1 .  Start  a  new  estimate 

2.  Update  size  estimate 

3.  Update  technology  assumptions 

4.  Update  schedule  assumptions 

5.  Update  staffing  assumptions 

6.  Update  labor  rate  and  FTE  assumptions 

7.  Time  now  calibration 

8.  Communicate  the  results 

9.  Re-Baseline  the  project 
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Example  Project:  Metrics  Charts 

Update  Size  Estimate 


Status  dashboard  -  Unit:  Go  Around 

ESS 

ppmc  sch  ed  u  1  e  Acco  m  pi  ish  me  nts 

Performance  Indices 

n 

%  of  Base  re  Plain 


CPU  1 


E  as  t  I  IK  A"  Compli-i  .55777  Hew;  34.44  'flQBTiPc 


Defects  Tracking 


CPU  1 


Index 

3,00 


CPU  1  (Cumulative) 


0  CF1 


ASR1 


jlI 


1-05  305  505  705  305  11-05  1-OS  30>S  5-OS  70>S  Date 


Health  Status  Indicator 


Schedule 

Variance 

Time 

Variance 

Cost 

Variance 

Size 

Growth 

Oefects 

E-ETTEP 

WORSE 

BETTER 

NO  CHANGE 

WORSE 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 
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Example  Project:  Metrics  Charts 

Update  Technology  Assumptions 


^  Status  dashboard  -  Program:  CPU  t 

BE® 

PPMC  Schedule  Accomplishments 

Performance  Indices 

n 

%  of  Bsse  re  Plain 


CPU  1 


E  as  t  I  IK  A"  Compli-i  .55777  Hew;  34.44  'flQBTiPc 


Defects  Tracking 


CPU  1 


Index 

3,00 


CPU  1  (Cumulative) 


0  cpi 


ASR1 


jlI 


1-05  305  505  705  305  11-05  1-OS  30>S  5-OS  70>S  Date 


Health  Status  Indicator 


Schedule 

Variance 

Time 

Variance 

Cost 

Variance 

Size 

Growth 

Oefects 

BETTER 

WORSE 

BETTER 

NO  CHANGE 

WORSE 

N/A 

N/A 

N/A 

N/A 

N/A 

N/-A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 
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Example  Project:  Metrics  Charts 

Update  Staffing  Assumptions 


Status  Dashboard  -  Program:  CPU  1 


ppmc  sch  ed  li  I  e  Acco  m  pi  ish  me  nts 


%  of  Bsse  re  Plain 


CPU  1 


E  as  t  T4  A"  5*777  HtosMs:  34.44  Worths 


Defects  Tracking 


CPU  1 


Performance  Indices 


Schedule 

Variance 

Time 

Variance 

Cost 

Variance 

Size 

Growth 

Defects 

BETTER 

WORSE 

BETTER 

NO  CHANGE 

WORSE 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 
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Example  Project:  Metrics  Charts 

Re-Baseline  the  Project 


Status  Dashboard  -  Program:  CPU  1 


ppmc  sch  ed  li  I  e  Acco  m  pi  ish  me  nts 


%  of  Basel  ne  Plan 
LQO-r 


CPU  1 


1-06  4-05  7-05  1005  1-06  4-06  7-06  1006  1-07  407  707  1007 

BzdneAl  Cui4Mii.MMa  mms;34jU  XtaBTlta. 


Defects  Tracking 


CPU  1 


Performance  Indices 


CPU  1 


Schedule 

Variance 

Time 

Variance 

Cost 

Variance 

Size 

Grc-irth 

Defects 

BETTER 

BETTER 

BETTER 

NO  CHANGE 

WORSE 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 
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Example  Project:  Metrics  Charts 

at  Code  &  Unit  Test  Complete 


IS  Status  Dashboard  -  Program:  CPU  1 


□0® 


P PMC:  sch edule  Aeco m pi ish m e nts 


1-G5  4-05  7-05  LOOS  1-06  4C6  7-06  1006  L-07  4-07  7-07  1007  L-G» 

EfAEUCOTVldcttiZKZ  HQflMSv  3T  XVaSTlte 


Performance  Indices 


CPU  1  (Cumulative) 


0  cp 


ASP* 


jlI 


1-05  7-05  1-OS  7-OS  1-07  7-07  Date 


Defects  Tracking 


Health  _Status  indicator 


Schedule 

Variance 

Time 

Variance 

Cost 

Variance 

Size 

Growth 

Defects 

BETTER 

WORSE 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 
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Example  Project:  Metrics  Charts 

Update  Staffing  Assumptions 


Status  Dashboard  -  Program:  CPU  1 


ppmc  sch  ed  li  I  e  Acco  m  pi  ish  me  nts 


%  of  Basel  ne  Plan 
LQO-r 


CPU  1 


105  4-05  7-05  1005  105  405  705  1005  107  407  707  1007  1-08 
Basina  M  Corn  pic:*:  4SJ-EJ  Soura-:  37.53  Xfcertta 


Defects  Tracking 


CPU  1 


Performance  Indices 


CPU  1  (Cumulative) 


jlI 


Health  Status  Indicator 


CPU  1 


0  CF1 


ASP* 


1-05  7-05  1-OS  7-OS  1-07  7-07  Date 


Schedule 

Variance 

Time 

Variance 

Cost 

Variance 

Size 

Growth 

Defects 

BETTER 

WORSE 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 
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Example  Project:  Metrics  Charts 

Re-Baseline  the  Project 


Status  Dashboard  -  Program:  CPU  1 


ppmc  sch  ed  li  I  e  Acco  m  pi  ish  me  nts 


%  of  Basel  ne  Plan 
LQO-r 


CPU  1 


1-05  4-05  7-05  1005  1-05  4-05  7-05  1005  107  4-07  7-07  1007  1-Q« 

E24k  Aft  Compl*-*:  Htow=::  3E.32 


Defects  Tracking 


CPU  1 


Performance  Indices 


Health  Status  Indicator 


CPU  1 


Schedule 

Variance 

Time 

Variance 

Cost 

Variance 

Size 

Grc-irth 

Defects 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 
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Example  Project:  Metrics  Charts 

at  Component  Int.  &  Test  Complete 


Status  Dashboard  -  Program:  CPU  1 


ppmc  sch  ed  li  I  e  Acco  m  pi  ish  me  nts 


%  of  Base  re  Plan 
1GQ-,- 


CPU  1 


105  4-05  7-05  1005  105  406  706  1006  107  407  707  1007  1-Q« 
BaMHiKAI  Constate  467X4  HSmms-:  3333  'tort to 


Defects  Tracking 


CPU  1 


Performance  Indices 


Health  Status  Indicator 


CPU  1 


Schedule 

Variance 

Tims 

Variance 

Cost 

Variance 

Size 

Grc-irth 

Defects 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 
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9  Software  projects  fail  more  often  than  not 
9  Project  success  Good  management 
9  Measurement  objectifies  management 
o>  Software  projects  are  governed  by  dynamic  properties 

•  Properties  currently  accounted  for  in  the  Project  Planning  process 

•  Properties  should  also  be  accounted  for  in  the  Project 
Monitoring  and  Control  process 

o>  Project  Monitoring  Performance  Measurement 
o>  4-D  Earned  Value  objectifies  progress 
»  Project  Control  Control  Limits 
»  Re-Baselining  Performance-Based  Forecasting 

9  Communication  is  essential  to  successful  project  management 
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Abstract.  Performance  Measurement  (an  integral  part  of  Earned  Value 
Management  (EVM))  has,  over  at  least  the  last  two  decades,  become  a 
gold  standard  process  (i.e.,  best  practice)  for  monitoring  and  controlling 
the  progress  of  software  development  projects.  It  employs  the 
fundamental  measurement-based  command/feedback  principals  of  control 
theory  to  increase  the  probability  that  a  project’s  actual  performance 
matches  its  expected  (planned)  performance;  i.e.,  that  a  project  is 
delivered  on  time  and  within  budget  or,  at  least,  that  there  is  an  early 
warning  of  looming  disaster.  This  process  is  generally  well-understood  by 
project  managers  and  reasonably  well  supported  by  commercially- 
available  tools.  Experience  with  this  process  suggests  an  opportunity  for 
significant  process  improvement  by  including  established  estimation 
methodology  and  algorithms  as  part  of  the  forecasting  and  re-baselining 
activities  performed  during  the  project  monitoring  and  control  process. 

This  paper  first  reviews  the  fundamentals  of  software  project  management 
and  of  Performance  Measurement  (including  some  proposed  extensions 
to  the  notion  of  earning  value)  and  then  proposes  a  process  called 
Parametric  Project  Monitoring  and  Control  (PPMC)  whereby  accepted 
algorithms  currently  used  for  software  cost  and  schedule  estimation  during 
the  project  planning  process  are  incorporated  into  the  forecasting  and  re¬ 
baselining  processes  to  yield  a  more-realistic  time-range  prediction  of  the 
project’s  cost  and  duration. 
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Introduction 


Purpose 

The  purpose  of  this  paper  is  to  first  review  the  fundamentals  of  software  project 
management  and  of  Performance  Measurement  (including  some  proposed  extensions 
to  the  notion  of  earning  value)  and  then  to  propose  a  process  called  Parametric  Project 
Monitoring  and  Control  (PPMC)  whereby  accepted  algorithms  currently  used  for 
software  cost  and  schedule  estimation  during  the  project  planning  process  are 
incorporated  into  the  Estimate  at  Completion  (EAC)  calculation  to  yield  a  more-realistic 
time-range  prediction  of  the  project’s  cost  and  duration. 

Scope 

This  paper  applies  to  the  project  management  aspects  of  the  software  development 
process;  particularly  to  those  Level  2  process  areas  referred  to  in  the  Carnegie  Mellon 
University  Software  Engineering  Institute’s  (SEI)  Capability  Maturity  Model®  Integration 
(CMMI™)  for  Project  Planning,  Project  Monitoring  and  Control,  and  Measurement  and 
Analysis  [1].  While  the  scope  and  focus  of  this  paper  is  the  software  development 
process,  one  could  imagine  how  these  ideas  can  be  readily  applied  to  hardware  and 
system  development  as  well. 

Background 

Performance  Measurement  (an  integral  part  of  Earned  Value  Management  (EVM))  has, 
over  at  least  the  last  two  decades,  become  a  gold  standard  process  (i.e.,  best  practice) 
for  monitoring  and  controlling  the  progress  of  software  development  projects.  It  employs 
the  fundamental  measurement-based  command/feedback  principals  of  control  theory  to 
increase  the  probability  that  a  project’s  actual  performance  matches  its  expected 
(planned)  performance;  i.e.,  that  a  project  is  delivered  on  time  and  within  budget  or,  at 
least,  that  there  is  an  early  warning  of  looming  disaster.  This  process  is  generally  well- 
understood  by  project  managers  and  reasonably  well  supported  by  commercially- 
available  tools.  E xperience  with  this  process  suggests  an  opportunity  for  significant 
process  improvement  by  including  established  estimation  methodology  and  algorithms 
as  part  of  the  forecasting  and  re-baselining  activities  performed  during  the  Project 
Monitoring  and  Control  process. 


Software  Project  Management 

The  primary  purpose  of  the  software  project  management  process  is  to  ensure  that  its 
associated  software  development  project  is  successful.  Success,  within  this  context,  is 
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assumed  to  mean  achieving  or  exceeding  expectations;  i.e.,  success  occurs  when  the 
actual  outcome  matches  (within  a  reasonable  tolerance)  the  expected  outcome  [4], 

The  above  definition  of  success,  by  virtue  of  its  reference  to  expectations,  implies  the 
need  for  some  sort  of  roadmap  or  plan;  i.e.,  some  sort  of  description  of  these 
expectations  in  terms  of  who,  what,  when,  where,  how,  and  why.  It  follows,  then,  that 
one  of  the  software  project  management  process’s  primary  activities  should  be 

planning. 

Assuming  correspondence  between  expectations  and  some  sort  of  project  plan,  we  can 
now  assert  that  success  occurs  when  the  actual  outcome  matches  this  plan  (within 
some  reasonable  tolerance).  Ensuring  that  the  actual  outcome  matches  the  plan  must, 
therefore,  be  of  primary  concern  to  the  software  project  management  process.  Ensuring 
this  match  implies  influencing  the  actual  outcome  and/or  changing  the  plan;  therefore, 
one  of  the  software  project  management  process’s  primary  activities  should  be 
controlling. 

Influencing  the  actual  outcome  is  typically  achieved  by  careful  initialization,  direction, 
and  correction  of  the  software  development  process  and  of  the  environment  in  which  it 
is  performed.  Since  the  software  development  process  is  fueled  primarily  by  labor 
(people),  one  of  the  software  project  management  process’s  primary  activities  should  be 
resourcing.  Since  software  development  is,  indeed,  a  process,  it  involves  methods, 
skills/expertise,  tools,  and  task  flow.  Therefore,  the  set  of  software  project  management 
process  primary  activities  should  also  include  organizing,  training,  and  equipping. 

Finally,  once  the  project  is  complete,  we  must  deliver  the  product,  determine  whether  or 
not  the  project  was  a  success,  and  learn  from  all  that  was  measured  and  experienced 
during  the  project.  We  therefore  suggest  that  one  of  the  software  project  management 
process’s  primary  activities  should  be  transitioning;  transitioning  the  project’s  product  to 
the  consumer  and  transitioning  the  project’s  knowledge  and  experience  to  the  next 
project(s). 

A  useful  mental  device  for  remembering  the  above-described  software  project 
management  process  primary  activities  is  the  fact  that  the  first  letter  of  each  activity 
name  forms  the  word  PROTECT. 

m  Planning- estimating,  scheduling 

■  Resourcing  -  interviewing,  hiring,  motivating 

■  Organizing  -establishing  interpersonal  communication  paths  and  rules, 
mapping  resources  to  tasks 

■  Training -teaching,  mentoring 

■  Equipping  -  acquiring  and  allocating  equipment,  tools,  materials,  supplies, 
products  etc. 

■  Controlling  -  directing,  measuring,  correcting  and/or  replanning 


October  24,  2005 


Page  3  of  36 


Rev  D 


■  Transitioning  -delivering,  reviewing,  analyzing,  archiving 

The  software  project  management  process,  by  ensuring  project  success,  is  protecting 
the  sponsoring  organization’s  investment  in  the  project.  Or,  perhaps  (I  can’t  resist 
injecting  a  little  gallows  humor  here),  it  is  protecting  the  software  project  manager’s  job. 

Mapping  Software  Project  Management  to  the  CMMI 

If  we  slightly  reorganize  our  PROTECT  model  such  that  the  planning  and  controlling 
activities  each  subsume  the  appropriate  aspects  of  resourcing,  organizing,  training, 
and  equipping,  then  we  are  left  with  three  top-level  activities:  planning,  controlling,  and 
transitioning.  These  three  activities  can  be  one-to-one  mapped  with  the  Level  2  CMMI™ 
process  areas  for  Project  Planning,  Project  Monitoring  and  Control,  and 
M  easurement  and  Analysis.  Figure  1  depicts  a  proposed  context  model  that  illustrates 
the  relationships  between  these  process  areas. 


Figure  1 :  Software  Project  Management  Process  Areas  Context 

The  following  subordinate  sections  represent  an  abbreviated  summary  of  each  software 
project  management  related  Level  2  CMMI™  process  area.1 


Project  Planning 

The  purpose  of  Project  Planning  is  to  establish  and  maintain  plans  that  define  project 
activities. 


1 1nformation  in  each  subordinate  section  was  extracted  from  (CMMI®  Product  Team  2002).  Note  that  the  institutionalization-related 
goals/practices  have  been  omitted  for  brevity. 
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Establish  Estimates 

Estimate  the  Scope  of  the  Project 

Establish  Estimates  of  Work  Product  and  Task  Attributes 

Define  Project  Life  Cycle 

Determine  Estimates  of  Effort  and  Cost 

Develop  a  Project  Plan 

Establish  the  Budget  and  Schedule 

Identify  Project  Risks 

Plan  for  Data  Management 

Plan  for  Project  Resources 

Plan  for  Needed  Knowledge  and  Skills 

Plan  Stakeholder  Involvement 

Establish  the  Project  Plan 

Obtain  Commitment  to  the  Plan 

Review  Plans  that  Affect  the  Project 
Reconcile  Work  and  Resource  Levels 
Obtain  Plan  Commitment 

Project  Monitoring  and  Control 

The  purpose  of  Project  Monitoring  and  Control  is  to  provide  an  understanding  of  the 
project’s  progress  so  that  appropriate  corrective  actions  can  be  taken  when  the  project’s 
performance  deviates  significantly  from  the  plan. 

Monitor  Project  Against  Plan 

Monitor  Project  Planning  Parameters 
Monitor  Commitments 
Monitor  Project  Risks 
Monitor  Data  Management 
Monitor  Stakeholder  Involvement 
Conduct  Progress  Reviews 
Conduct  Milestone  Reviews 
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Manage  Corrective  Action  to  Closure 

Analyze  Issues 

Take  Corrective  Action 

Manage  Corrective  Action 

Measurement  and  Analysis 

The  purpose  of  M  easurement  and  Analysis  is  to  develop  and  sustain  a  measurement 
capability  that  is  used  to  support  management  information  needs. 

Align  Measurement  and  Analysis  Activities 

Establish  Measurement  Objectives 
Specify  Measures 

Specify  Data  Collection  and  Storage  Procedures 
Specify  Analysis  Procedures 

Provide  Measurement  Results 

Collect  Measurement  Data 
Analyze  Measurement  Data 
Store  Data  and  Results 
Communicate  Results 

Fundamentals  of  Software  Development 

Software  Development  Taxonomy 

Terms  defined:2 

Abstraction  -  A  representation  of  an  idea  or  concept  expressed  in  a  particular  medium 
or  language. 

Desire  -  A  want  or  need. 

[Software]  Requirements  -  An  abstraction  of  a  desire  for  which  computer  technology 
is  thought  to  be  a  viable  solution;  the  essence  of  a  software  product. 

Software  -  An  abstraction  of  a  desire  expressed  as  instructions  and  data  in  a  form  that 
can  be  acted  upon  by  a  computer. 


2  Term  definitions  extracted  from  [4]. 
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Process  -A  set  of  actions  or  operations  conducing  to  an  end  [3]. 

Software  Development  Process  -  A  generalized  set  of  related  activities  that  transform 
desires  into  software. 

Software  [Development]  Project  -  A  specific  instance  of  a  software  development 
process. 

Software  Product  -  The  primary  (deliverable)  result  of  a  Software  Development 
Project;  the  implementation  of  a  software  product. 

Software  Development  Process  Context 

Figure  2  depicts  the  context  of  a  software  development  process;  i.e.,  how  it  interfaces 
with  its  environment.  All  instances  of  software  development  processes  seek  to 
transform  software  requirements  into  a  software  product.  To  accomplish  this 
transformation,  they  consume  energy  in  the  form  of  labor  (people  doing  work)  from 
project  initiation  to  project  completion.  Since  no  software  development  process  is  a 
perfect  machine,  it  produces  some  amount  of  waste  or  entropy  (undesired  byproducts). 


Desire 


Labor 


Technology 


^Finish 


Software 


Figure  2:  Software  Development  Context3 


3  Figure  reprinted  with  permission  from  Michael  A.  ROSS  Consulting  &  Training.  All  rights  reserved. 
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Measuring  the  Software  Development  Process 

The  key  to  effectively  and  efficiently  measuring  the  software  development  process  is  to 
pick  measures  that  quantify  the  process’s  connections  to  its  surrounding  environment. 
Just  about  any  core  set  of  software  development  process  measures  will  include  the 
following: 

Size  -  An  abstraction’s  mass,  inertia,  bigness  (as  it  directly  relates  to  the  work  that  must 
be  done). 

Duration  -  The  elapsed  calendar  time  between  process  initiation  and  process 
completion. 

Effort  Cost,  Staffing  -  People  doing  work  during  the  software  development  process 
and  their  associated  cost,  over  elapsed  calendar  time. 

Quality  -  Defect  discovery  and  removal  over  elapsed  calendar  time. 


Effective  Technology 
(coefficient) 


_  Staffing  _ 
(people,  time) 

Effort 

(person-months) 

Labor 

^ _  Cost  , 

(currency) 

Technology 


Size 

(work  units) 

A 


Desire 


Start 


Size 

(work  units) 


Finish 


Time 

(calendar  months) 


Figure  3:  Software  Development  Process  Context  with 
Measurement4 


A  strong  indication  of  the  usefulness  of  these  measures  is  the  fact  that  they  address  the 
most  frequently  asked  questions  about  software  development  projects: 

How  big  will  the  product  be  when  delivered? 

How  long  is  it  going  to  take  ? 

How  many  people  will  be  needed  and  when? 


4  Figure  reprinted  with  permission  from  Michael  A.  ROSS  Consulting  &  Training.  All  rights  reserved. 
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How  much  will  it  cost? 

How  reliable  will  the  product  be  when  delivered? 


Progress  Defined 


Transforming  Desires  to  Software 

The  fundamental  goal  (i.e.,  the  root  of  progress)  associated  with  the  software 
development  process  is  the  transformation  of  a  mass  of  knowledge  articulated  as  a 
desire  into  a  mass  of  knowledge  articulated  as  software.  While  this  transformation  may, 
in  actuality,  exist  in  a  space  of  many  dimensions  and  directions  (bordering  on  chaotic), 
we  choose  to  focus  on  a  single  dimension  we  will  call  progress,  the  axis  or  direction  of 
which  we  will  call  s . 

Progress  Position  and  Change 

Within  dynamic  systems,  the  concept  of  position  is  typically  used  to  quantify  location 
within  some  space  relative  to  some  selected  reference  point.  Position  is  typically 
represented  as  a  vector  quantity;  i.e.,  it  consists  of  both  magnitude  and  direction  from 
some  given  reference  point  or  point  of  origin.  We  have  already  assumed  that  our 
system  has  only  one  relevant  dimension  which  we  have  named  progress.  Within  the 
progress  dimension,  we  will  establish  the  initial  position  of  the  mass  of  knowledge 
articulated  as  a  desire  (the  initial  state  of  the  transformation)  as  being  the  point  of  origin 
of  measurement  and  will  measure  position  relative  to  this  point  in  the  5  direction  (along 
the  5  axis).  We  will  further  assume  that  our  system  is  restricted  to  the  positive  side  of 
the  s  axis;  i.e.,  we  will  assume  the  notion  of  negative  position  to  be  undefined.5 

Normalized  Earned  Value 

Since  our  system  is  a  transformation  from  one  tangible  state  (the  desire)  to  another 
tangible  state  (the  software)  with  intermediate  states  being  somewhat  intangible,  we 
choose  unity  (1  or  100%  complete)  to  represent  the  position  of  the  transformation  at 
completion.  As  previously  stated,  the  initial  position  of  the  transformation  is  assumed  to 
be  zero  (0%  complete).  We  define  any  value  that  represents  a  specific  position  on  the 
continuum  of  intermediate  position  values  as  normalized  earned  value.  In  other  words,  if 
a  project’s  position  in  the  direction  of  progress  is  0.5  (50%  complete),  then  that  project’s 
normalized  earned  value  is  said  to  be  0.5  or  50%.  If  a  project  has  been  assigned 


5  The  author  acknowledges  that  this  assumption  is  open  to  challenge;  i.e.,  that  it  might  be  necessary  to  account  for  the  notion  of  project 
digression  beyond  the  initial  state.  The  author,  however,  chooses  to  take  the  more  positive  viewpoint  that  what  appears,  on  the  surface,  to  be 
digression,  is  actually  part  of  the  necessary  learning  process  that  is  key  to  any  development  activity.  In  other  words,  while  it  might  not  be 
obvious  at  the  time,  you  are  always  better  off  today  than  you  were  yesterday. 
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(estimated  to  be)  some  budgeted  monetary  value  (measured  in  units  of  currency)  or 
some  budgeted  effort  value  (measured  in  units  of  labor),  then  the  project’s  earned  value 
may  be  expressed  as  the  product  of  its  normalized  earned  value  and  its  total  budgeted 
monetary  value  or  total  budgeted  effort  value  respectively. 

Closely  related  to  position  is  the  concept  of  displacement,  which  is  typically  used  to 
quantify  some  change  in  position.  It  too  is  typically  represented  as  a  vector  quantity. 
Displacement  can  be  described  mathematically  as  [2]: 

,t2)  =  r(t2)-r(t1)  Eqn.1 

where: 

tx  Some  initial  point  in  time. 

t2  Some  final  point  in  time  where  t2  >  tx . 

r(rj)  The  initial  position  vector;  the  position  vector  at  time  r, . 

r (t2)  The  final  position  vector;  the  position  vector  at  time  t2 . 

\r{tvt2)  The  displacement  vector  from  the  initial  position  to  the  final  position. 

Within  the  progress  dimension  s ,  the  magnitude  of  the  displacement  vector,  \\r(tvt2)\ , 
quantifies  a  change  in  normalized  earned  value  from  the  position  at  some  time  t{  to  the 
position  at  some  later  time  t2 . 

A  Review  of  Performance  Measurement  Relationships 

This  section  contains  a  brief  definition  and  description  of  relevant  Performance 
Measurement  relationships.  Considering  the  three  forms  of  earned  value  introduced  in 
the  preceding  section  ( monetary ,  effort,  and  normalized ),  we  include  equations  for  each 
of  these  three  unit  systems  ( currency ,  labor,  and  %  of  B AC)  where  meaningful  to  do 
so. 

Budget  at  Completion  (BAC) 

The  starting  point  or  anchor  measure  for  Performance  Measurement  is  the  Budget  at 
Completion  (BAC).  This  measure  is  a  primary  output  of  the  estimation  process  and 
represents  what  is  expected  to  be  spent  by  the  project.  The  plan  that  is  associated  with 
this  value  is  generally  referred  to  as  the  Baseline  Plan  or  simply  the  Baseline.  Note  that 
in  the  normalized  unit  system,  BAC  is  always  assumed  to  be  unity  (1  or  100%).  Note 
also  that  the  currency  and  labor  variants  of  this  value  may  change  during  the  course  of 
a  project  as  the  result  of  re-planning  or  re-baselining. 
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For  units  of  currency: 


BAC%  =  X  Activity.  Planned  Cost 

i= 1 

Where: 

n  The  total  number  of  activities  in  the  project 

For  units  of  labor: 

n 

BACl  =  y  Activity;  Planned  Effort 

;=i 

Where: 

n  The  total  number  of  activities  in  the  project 

For  a  normalized  (%  of  BAC)  measurement  system: 

BAC%=\Ar(tm,tfm,h)\  =  l  =  100% 

Three  Fundamental  Cost  of  Work  Measures 

Performance  Measurement  requires  that,  for  a  given  amount  of  elapsed  calendar  time 
t ,  we  know: 

Flow  much  we’ve  planned  to  spend, 

Flow  much  we’ve  earned  (progressed),  and 
Flow  much  we’ve  actually  spent. 

Budgeted  Cost  of  Work  Scheduled  (BCWS) 

Value  of  the  work  planned  to  be  done  -  also  known  as  the  Planned  Value  (PV). 

For  units  of  currency: 

BCWS$  (0  =  X  Activity,.  Planned  Cost  Et>n- 5 

j= i 

Where: 


Eqn.2 


Eqn.  3 


Eqn.  4 
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Activity 7  Must,  in  order  to  count,  be  planned  for  completion  by  time  t . 
n  Total  number  of  activities  in  the  project. 


For  units  of  labor: 

n 

BCWSl  (?)  =  7  Activity,  Planned  Effort 

7=1 

Where: 

Activity .  Must,  in  order  to  count,  be  planned  for  completion  by  time  t . 
n  Total  number  of  activities  in  the  project. 


For  a  normalized  (%  of  BAC)  measurement  system: 


BCWS%  (?)  = 


BCWS$(?)_BCWSl(?) 
BAC$  BACl 


Budgeted  Cost  of  Work  Performed  (BCWP) 

Value  of  the  work  earned  (accomplished)  -  also  known  as  the  Earned  Value  (EV) 
For  units  of  currency: 

m 

BCWP$  (?)  =  X Completed  Activity;  Planned  Cost 

;= 1 

Where: 

Activity,  Must,  in  order  to  count,  be  complete  by  time  ? . 
m  Total  number  of  complete  activities  in  the  project. 

For  units  of  labor: 

m 

BCWP£  (?)  =  7 Completed  Activity,  Planned  Effort 

i= i 

Where: 

Activity,  Must,  in  order  to  count,  be  complete  by  time  ? . 
m  Total  number  of  complete  activities  in  the  project. 


Eqn.  6 


Eqn.7 


Eqn.  8 


Eqn.  9 
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For  a  normalized  (%  of  BAC)  measurement  system: 


BCWP%(<)  =  |Ar(0,<)|  = 


BCWP,(t) 

BAC% 


BCWP L(t) 
BACl 


Actual  Cost  of  Work  Performed  (ACWP) 

Actual  amount  spent  to  accomplish  the  work  -  Actual  Cost  (AC). 
For  units  of  currency: 

n 

ACWP$  (/)  =  ^  Activity ^  Actual  Accumulated  Cost 

k= 1 

Where: 

n  Total  number  of  activities  in  the  project. 


For  units  of  labor: 

n 

ACWPl  (t)  =  Ji  Activity k  Actual  Accumulated  Effort 

k=\ 

Where: 

n  Total  number  of  activities  in  the  project. 


For  a  normalized  (%  of  BAC)  measurement  system: 


ACWP,,  (?) 


\l  'A  I’  (f )  _  ACWP,  (?) 
BACS  BAC  L 


Fundamental  Variances 

Cost  Variance  (CV) 

Difference  between  earned  and  spent.  Positive  values  are  favorable, 
are  unfavorable. 

For  units  of  currency: 

CV$  {t)  =  BCWP$  (t)  -  ACWP$  (t) 


Eqn.10 


Eqn.11 


Eqn.  12 


Eqn.  13 


negative  values 


Eqn.  14 


October  24,  2005 


Page  13  of  36 


Rev  D 


For  units  of  labor: 


CVt(»)  =  BCWP,.(»)-ACWP,,(r) 


For  a  normalized  (%  of  BAC)  measurement  system: 

CV%  (r )  =  BCWP%  (»)  -  ACWP%  (r ) 


Eqn.  15 


Eqn.  16 


Schedule  Variance  (SV) 

Difference  between  earned  and  planned.  Positive  values  are  favorable,  negative  values 
are  unfavorable. 

For  units  of  currency: 

SV$  (t)  =  BCWP$  (0  -  BCWS$  (t)  Eqn.  17 

For  units  of  labor: 

SVL  (f )  =  BCWPl  (f )  -  BCWSl  (f )  Eqn.  18 

For  a  normalized  (%  of  BAC)  measurement  system: 

SV%  (t)  =  BCWP%  (r)-BCWS%  (f)  Eqn.  19 


Additional  Variances 

Budget  Variance  (BV) 

Difference  between  planned  and  actual.  Positive  values  are  favorable,  negative  values 
are  unfavorable. 

For  units  of  currency: 

BV$  (f)  =  BCWS$  (0-  ACWP$  (0  Eqn.  20 

For  units  of  labor: 
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bv£  (?)  =  bcwsl  (?)  -  acwpl  (?) 


Eqn.  21 


For  a  normalized  (%  of  BAC)  measurement  system: 

BV%  (»)  =  BCWS%  (()- ACWP%  (<) 


Eqn.  22 


Time  Variance  (TV) 

Difference  in  elapsed  calendar  time  between  time  now  and  time  where  plan  function 
equals  earned  function  evaluated  at  time  now.  Positive  values  are  favorable,  negative 
values  are  unfavorable. 

For  units  of  elapsed  calendar  time: 

TV,  (0  =  ^bcwSj  =bcwp$ (?)  ~t  =  ^BCwst=Bcwp L(t)  ~  t  =  ^bcws%=bcwp%(( )  —  ^  Eqn.  23 

For  a  normalized  (%  of  BAC)  measurement  system: 

-rw  I *\  —  ^BcwSj=Bcwp$(f)  ^bcwsl=bcwp L(t)  _  ^bcws%=bcwp %(t)  ~ ^  Eqn.  24 

1V%P)  =  —  =—  =  — 

*BCWS$=BCWP$(f)  bCWSl=BCWPl(?)  *BCWSo/o=BCWP0/o(f) 


Fundamental  Performance  Indices 


Cost  Performance  Index  (CPI) 


Amount  earned  to  amount  spent  ratio;  cost  efficiency  achieved  from  project  start  to  time 
?.  Values  greater  than  unity  are  favorable,  values  less  than  unity  are  unfavorable. 


CPI(?) 


BCWP$  (?) 
ACWP$  (?) 


BCWPl(?)_  BCWP%(?) 
ACWPl(?)_  ACWP0/o(?) 


Eqn.  25 


Schedule  Performance  Index  (SPI) 

Amount  earned  to  amount  planned  ratio;  schedule  efficiency  achieved  from  project  start 
to  time  ?.  Values  greater  than  unity  are  favorable,  values  less  than  unity  are 
unfavorable. 
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SPI(f)  = 


BCWP$  (t)  _  BCWPl  (t) 
BCWS$ (t)  ~  BCWS L(t) 


BCWPX(Q 
BCWS„/o  ( t ) 


Eqn.  26 


Additional  Performance  Indices 
Budget  Performance  Index  (BPI) 

Amount  planned  to  amount  spent  ratio;  budget  efficiency  achieved  from  project  start  to 
time  t.  Values  greater  than  unity  are  favorable,  values  less  than  unity  are  unfavorable. 

BCWS$(Q  s  BCWS£(r)  _  BCWS%(0  Eqn.  27 

1  ’  ACWP$  (r)  ACWP,  (r)  ACWP0/o(r) 


Time  Performance  Index  (TPI) 

Time  efficiency  achieved  from  project  start  to  time  t .  Values  greater  than  unity  are 
favorable,  values  less  than  unity  are  unfavorable. 


TPI(r)  = 


_  ^BCWSj=BCWPj(0  _  ^BCWSt=BCWI\(r)  _  ^BCWS%=BCWP%(/) 


Eqn.  28 


Composite  Performance  Index  (XPI) 

Combined  cost  and  schedule  efficiency  achieved  from  project  start  to  time  t.  Values 
greater  than  unity  are  favorable,  values  less  than  unity  are  unfavorable. 

XPl(r)  =  ^CPl(r)SPl(r)  Eqn.  29 


Status  and  Forecasting  Metrics 
Percent  Complete 

Percent  Complete  =  BCWP% 
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Percent  Spent 


Percent  Spent  =  ACWP% 


Eqn.  31 


Estimate  at  Completion  (EAC) 

Expected  (predicted)  actual  cost  value  when  normalized  earned  value  reaches  100%. 

EAC  Based  on  Cost  Variance  (Basic) 

For  units  of  currency: 

EACbasic$  (t)  =  BAC%  -  CV$  (f )  Eqn.  32 

For  units  of  labor: 

EACbasicL  (f )  =  BACL  -  CVL  ( t )  Eqn.  33 

For  a  normalized  (%  of  BAC)  measurement  system: 

EACbasie,  (t)  =  BAC„  —  CV%  (t)  Eqn.  34 


EAC  Based  on  Cost  Performance 


For  units  of  currency: 


EACcpij  (t) 


BAC% 

CPI(r) 


Eqn.  35 


For  units  of  labor: 


EACcpiL  (t) 


BAC L_ 
CPl(r) 


Eqn.  36 


For  a  normalized  (%  of  BAC)  measurement  system: 


EACcpi%  (?)  = 


BAC% 

CPl(f) 


Eqn.  37 
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EAC  Based  on  Composite  Performance 

For  units  of  currency: 


EACxpilW^ 


Eqn.  38 


For  units  of  labor: 


EAC  xpi,  (t)  =  ——fT 
LW  XPl(f) 


Eqn.  39 


For  a  normalized  (%  of  BAC)  measurement  system: 


EACxpi0/1  (t) 


BAC% 

XPl(f) 


Eqn.  40 


Estimate  to  Complete  (ETC) 

The  expected  additional  work  that  must  be  done  to  achieve  a  normalized  earned  value 
of  1 00%. 

For  units  of  currency: 

ETC$  (t)  =  BAC%-  BCWP$  (t)  Eqn.  41 


For  units  of  labor: 

ETC,  (?)  =  BACl  - BCWP,  (?)  Eqn.  42 

For  a  normalized  (%  of  BAC)  measurement  system: 

ETC%  (f )  =  BAC%  -  BCWP„/o  {t)  =  100%  -  BCWP%  {t)  Eqn.  43 


Variance  at  Completion  (VAC) 

Difference  between  planned  and  predicted.  Positive  values  are  favorable,  negative 
values  are  unfavorable. 


October  24,  2005 


Page  18  of  36 


Rev  D 


VAC  Based  on  Cost  Performance 

For  units  of  currency: 

VACcpi$  (t)  =  BAC$  - EACcpi$  (t)  Eqn.  44 

For  units  of  labor: 

VACcpiL  (r)  =  BACl  —  EACcpiL  (f)  Eqn.  45 

For  a  normalized  (%  of  BAC)  measurement  system: 

VACcpi0/o  ( /  )  =  BAC%  —  EAC  cpi,:  ( ? )  Eqn.  46 

VAC  Based  on  Composite  Performance 

For  units  of  currency: 

VACxpi$  ( t )  =  BAC$  -  EACxpi$  ( t )  Eqn.  47 

For  units  of  labor: 

VACxpiL  (t)  =  BACl  —  EACxpiL  (t)  Eqn.  48 

For  a  normalized  (%  of  BAC)  measurement  system: 

VACxpi%  (t)  =  BAC%  -  EACxpi%  (t)  Eqn.  49 


To  Complete  Performance  Index  (TCPI) 

Ratio  of  work  remaining  against  money  remaining  (efficiency  which  must  be  achieved  to 
complete  the  remaining  work  with  the  expected  remaining  money). 
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TCPI  Based  on  Cost  Variance  (Basic) 


TCPIbasic(r)  = 


Work  Remaining 
Cost  Remaining 


BAC$  -BCWP$  (?) 
EACbasic$  (?)  -  ACWP$  (?) 

BACl  -BCWPl  (?) 
EACbasicL  (?)  -  ACWP£  (?) 

BACVo  -BCWP0/o  (?) 
EACbasic%  (?)  -  ACWP%  (?) 


TCPI  Based  on  Cost  Performance 


TCPIcp(r)  = 


Work  Remaining 
Cost  Remaining 


BAC%  -  BCWP$  (t) 
EACcpi$  (t)  — ACWP$  (t) 
BACL-BCWVL(t) 
EACcpi£  (t)- ACWPl  (t) 
BAC%  -  BCWP%  (?) 
EACcpi%  (?)  -  ACWP%  (?) 


TCPI  Based  on  Composite  Performance 


TCPIxp(?)  = 


Work  Remaining 
Cost  Remaining 


BAC%  -BCWP$  (?) 
EACxpi$  (?)  -  ACWP$  (?) 

BACl  -  BCWPl  (?) 
EACxpiL  (?)  -  ACWPl  (?) 

BAC%  - BCWP%  (?) 
EACxpi%  (?)  -  ACWP%  (?) 


Average  Performance 

Average  of  work  accomplished  on  the  project  from  the  actual  start  time  tA 
as  the  earliest  time  t  where  ACWP„/o  (?')  >  0 ,  to  time  ? . 


For  units  of  currency: 


PA,  (r)  = 


BCWP,  (f) 

^  ^ A  start 


Eqn.  50 


Eqn.  51 


Eqn.  52 


,  defined 


Eqn.  53 
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For  units  of  labor: 


PAt(<)  = 


BCWP,(t) 

^  ^ A  start 


Eqn.  54 


For  a  normalized  (%  of  BAC)  measurement  system: 


PA.,W- 


BCWP%(f) 

^  ^ A  start 


Eqn.  55 


Average  Expected  Performance  to  Finish 

Average  of  the  work  which  must  be  accomplished  to  complete  the  project  at  the 
baseline  finish  time  tBL  finish ,  defined  as  the  earliest  time  t'  where  BCWS%  ( t‘ ')  =  1  =  100% . 

For  units  of  currency: 

— BCWP$  (?)  Eqn.  56 

^ BL_finish  ~  ^ 

For  units  of  labor: 

—  BAQ-BCWP^W  Eqn.  57 

L  ^  t  —t 

lBL_ finish  1 


For  a  normalized  (%  of  BAC)  measurement  system: 


PE  %(*)s 


BAC%  - BCWP„/o  (t) 


t 


BL  finish 


-t 


Eqn.  58 


Parametric  Project  Monitoring  and  Control  (PPMC) 


PPMC  Vision 

For  at  least  the  last  two  decades,  software  development  projects  have  benefited  from 
various  time-proven  techniques  for  estimating  the  duration  and  effort  of  software 
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development  projects.  Unfortunately  for  most  projects,  the  estimation  process  occurs 
only  once,  at  the  beginning  of  the  project,  at  a  time  when  the  least  is  known  about  what 
the  project  should  be.  In  the  very  worst  of  cases,  when  a  project  subsequently  gets  into 
trouble,  the  trouble  goes  unnoticed  until  the  inevitable  late  delivery  with  associated  cost 
overrun.  This  open-loop  process  behavior  is  illustrated  in  Figure  4  below. 


The  typical  first  step  in  trying  to  get  projects  under  control  is  the  introduction  of  a 
measurement  and  metrics  process  (e.g.,  Performance  Measurement)  which,  by  itself, 
supports  a  minimal  although  ad-hoc  project  control  capability  as  shown  in  Figure  5 
below. 


Next,  and  what  represents  the  current  status  quo,  is  the  introduction  of  tools  that 
schedule  tasks  and  allocate  resources  as  some  function  of  inter-task  dependencies, 
resource  availability,  and  priority.  These  tools  can  accept  measures  from  a  Performance 
Measurement  process  and  make  any  necessary  schedule  adjustments.  The  typical 
strategy  is  to  re-baseline  the  project  by  identifying  ljj>0n®|(l!yi0lete  tasks  in  the 
project  plan  as  a  subproject  and  slipping  the  schedule  of  the  entire  subproject  (as  a 
whole)  to  begin  on  the  current  date  tnow  (see  Figure  6  below). 
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The  problem  with  this  strategy  is  that  it  does  not  take  into  account  that  which  may  be 
causing  the  project  to  be  in  trouble  in  the  first  place;  namely  that  expected  performance 
(efficiency  or  productivity)  is  not  matching  actual  performance.  Instead,  we  are 
perpetuating  a  possibly  erroneous  assumption  about  performance  which  may  come 
back  to  bite  us  again.  We  therefore  propose  including  established  estimation 
methodology  and  algorithms  as  part  of  the  prediction  and  re-baselining  activities 
performed  during  the  Project  Monitoring  and  Control  process  and  refer  to  this  process 
as  Parametric  Project  Monitoring  and  Control  (PPMC).  The  idea  is  to  extend  the  scope 
of  software  development  project  estimation  to  include  situations  where  the  project  is 
already  under  way  and  where  some  project  actuals  already  exist.6 

In  order  to  realize  the  vision  of  PPMC,  we  propose  including  estimation  in  the  project 
monitoring  and  control  loop  (see  Figure  7  below). 


6  It  is  important  to  note  that  PPMC  is  not  intended  as  a  replacement  for  but  rather  as  complementary  to  the  Earned  Value  Management  (EVM) 
process. 


October  24,  2005 


Page  23  of  36 


Rev  D 


Extending  the  Notion  of  Quantifying  Progress  (Earning  Value) 

Within  the  realm  of  Performance  Measurement,  one  criticism  that  routinely  surfaces  is 
the  sometimes  arbitrary  manner  in  which  progress  or  value  (percent  complete)  is 
credited  to  activities  within  a  project  or  to  the  project  as  a  whole.  The  result  of  this 
arbitrariness  is  sometimes  referred  to  as  the  “90%  complete  90%  of  the  time”  syndrome. 
This  paper  proposes  a  four-dimensional  (4-D)  approach  for  assigning  progress  to  the 
development  of  each  program/application  (Computer  Software  Configuration  Item 
(CSCI))  that  is  part  of  the  project.  The  first  dimension  is  Software  Development  Life 
Cycle  ( SDLC )  primary  activity  completion  for  the  development  of  a  specific 
program/application  (CSCI).  Each  SDLC  primary  activity7,  in  turn,  is  assigned  progress 
according  to  a  weighted  combination  of  three  other  dimensions:  artifact  completion, 
milestone  completion,  and  defect  discovery/ removal. 

Program/Application  (CSCI)  Progress 

BCWP%  (r)  for  each  program/application  (CSCI)  is  computed  as  the  normalized 

weighted  sum  of  the  applicable  SDLC  primary  activities’  completion  percentage.  The 
weighting  scheme  is  directly  derived  from  the  effort-to-activity  allocation  defined  in  the 
project’s  baseline  plan. 

SDLC  Primary  Activity  Completion 

During  the  project  monitoring  and  control  process,  the  completion  percentage  for  each 
SDLC  primary  activity  is  computed  as  the  normalized  weighted  sum  of  its  artifacts 
completed,  its  milestones  completed,  and  its  associated  number  of  defects  discovered 
and  defects  removed. 

Artifact  Completion 

As  part  of  the  project  planning  process,  a  list  of  artifact  types  is  identified  and  each  type 
is  assigned  to  the  appropriate  SDLC  primary  activity.  Each  artifact  type  is  weighted 
according  to  its  relative  contribution  to  the  completion  of  its  associated  primary  activity. 
Additionally,  as  part  of  the  project  planning  process,  the  total  number  of  artifacts  of  each 
type  is  estimated  for  each  program/application  (CSCI).  These  totals  are  used  to 
determine  the  relative  weighted  contribution  of  each  completed  artifact  to  the  completion 
of  its  associated  SDLC  primary  activity. 

During  the  project  monitoring  and  control  process,  the  total  number  of  completed 
artifacts  of  each  type  are  periodically  and/or  aperiodically  measured  and  recorded. 


7  An  example  of  a  set  of  SDLC  primary  activities  is  System  Requirements  Design,  Software  Requirements  Analysis,  Preliminary  Design, 
Detailed  Design,  Code  &  Unit  Test,  Component  Integration  &  Test,  Program  Test,  System  Integration  thru  Operational  Test  &  Evaluation. 
There  are  various  SDLCs,  each  with  its  own  unique  set  of  primary  activities. 
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Milestone  Completion 

As  part  of  the  project  planning  process,  a  list  of  milestones  (events)  is  identified  and 
each  is  assigned  to  the  appropriate  SDLC  primary  activity.  Each  milestone  is  weighted 
according  to  its  relative  contribution  to  the  completion  of  its  associated  primary  activity. 

During  the  project  monitoring  and  control  process,  milestone  completion  state  is 
periodically/aperiodically  determined  and  recorded. 

Defect  Discovery  /  Removal 

As  part  of  the  project  planning  process,  an  estimate  is  made  for  each 
program/application  (CSCI)  of  its  total  theoretical  number  defects.  This  body  of  defects 
is  then  twice  distributed  over  elapsed  calendar  time;  once  according  discovery  and  once 
according  to  removal.  The  body  of  defects  is  also  proportionally  distributed  across  the 
set  of  SDLC  primary  activities  according  to  each  activity’s  corresponding  effort 
allocation.  The  total  number  of  defects  allocated  to  each  SDLC  primary  activity  is  used 
to  determine  the  relative  weighted  contribution  of  each  discovered  and  removed  defect 
to  the  completion  of  its  associated  SDLC  primary  activity. 

During  the  PPMC  process,  the  total  numbers  of  discovered  and  of  removed  defects  for 
each  SDLC  primary  activity  are  periodically/aperiodically  measured  and  recorded. 

At-a-Glance  Status  Indication  of  Measures  and  Metrics 

The  traffic  signal  metaphor  has  become  a  fairly  commonplace  strategy  to  indicate  Work 
Breakdown  Structure  (WBS)  individual  task  status  (red,  yellow,  green).  Since  status  can 
be  tracked  as  a  function  of  elapsed  calendar  time,  the  notion  of  status  is  often-times 
supplemented  by  the  inclusion  of  status  trend  (better,  no  change,  worse).  We  propose 
to  extend  this  strategy  to  include  several  measures  and  metrics  associated  with  each 
program/application  (CSCI)  that  are  managed  as  part  of  PPMC.  These  measures  and 
metrics  include:8 

Cost  Variance  and  Cost  Performance  Index 
Schedule  Variance  and  Schedule  Performance  Index 
Budget  Variance  and  Budget  Performance  Index 
Time  Variance  and  Time  Performance  Index 
Size  Growth  [4] 

Size  Uncertainty  Convergence  [4] 

Defect  Discovery  Variance 
Defect  Removal  Variance 


8  This  is  an  evolving  list  of  measures  and  metrics  and  is  expected  to  grow  as  experience  is  gained  applying  PPMC  to  real  projects. 


October  24,  2005 


Page  25  of  36 


Rev  D 


Control  Limits 

Earlier  in  this  paper  we  defined  success  to  mean  achieving  or  exceeding  expectations 
and  asserted  that  success  occurs  when  the  actual  outcome  matches,  within  a 
reasonable  tolerance,  the  expected  outcome.  In  the  preceding  section  we  proposed 
quantifying  measurement  and  metric  status  using  the  traffic  signal  metaphor  with  trend 
supplement.  In  order  to  realize  the  traffic  signal  metaphor  for  a  given  measure  or  metric, 
we  must  define  all  the  transitions  between  green  and  yellow  and  between  yellow  and 
red.  We  propose  to  call  these  transitions  control  limits  and  include  them  as  part  of  our 
PPMC  process  in  order  to  link  the  notion  of  reasonable  tolerance  to  the  notion  of  status. 

We  now  propose  the  following  definition  for  the  set  of  control  limits  associated  with  a 
particular  measure  or  metric  where  the  particular  measure  or  metric  can  be  expressed 
as  a  function  of  elapsed  calendar  time:9 

control_limit_set  =  upper_yellow _to _red_control_limit  Eqn  59 

upper_green  to  _  yellow _control_limit 
lower_green _to _  yellow _control_limit 
lower jyellow  _to  _  red_control_limit 


upper_yellow  _to  _  red_control_limit  = 

control_limit_segment 
[control_limit_segment]  K 

Eqn.  60 

upper _green _to _  yellow _control_limit  = 

control_limit_segment 
[control_limit_segment]  K 

Eqn.  61 

upper _yellow  _to  _  red_control_limit  = 

control_limit_segment 
[control_limit_segment]  K 

Eqn.  62 

upper_yellow  _to  _  red_control_limit  = 

control_limit_segment 
[control_limit_segment]  K 

Eqn.  63 

control_limit_segment  =  starting 

month  number 

Eqn.  64 

scale  _  factor 
offset 

Note  that  starting  _month  _number  =  0  indicates  both  a  null  control_limit_segment  and 
the  last  control, _limit_segment  in  a  list. 

The  control_limit_segment  function  u .  ( ti )  for  the  j'h  control_limit_segment  of  a  particular 
controljimit  in  a  controljimit _set  associated  with  a  particular  measure  or  metric  that  is 


9  The  author  acknowledges  that  the  “perfect”  control  limit  schema  would  allow  specification  as  any  function  of  elapsed  calendar  time.  The 
proposed  schema  represents  a  compromise  between  flexibility  and  computational  complexity. 
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a  function  of  elapsed  calendar  time  m(r,.)  and  evaluated  at  the  ith  month  into  the  project 
is  defined  as: 


uj{ti)  =  {aj)m{ti)  +  bj 


Eqn.  65 


Where: 

m  One  more  than  the  count  of  whole  months  in  the  range  tstan  to  tfinish 
where  me  ¥  . 

n  The  count  of  control  limit  segments  in  a  particular  control  limit  where 

ne  ¥  . 

i  Month  number  where  ie  ¥  n[l,m] . 

j  Control  limit  segment  index  where  je  0  *n[0,n] 

tt  Quantum  elapsed  calendar  time  from  tstan  to  i  months  into  the  project. 

cij  Scale  factor  of  the  f  control  limit  segment. 

bj  Offset  of  the  jth  control  limit  segment;  must  be  scaled  in  the  same 
units  as  is  m(r  ). 

m(ri.)  A  particular  measure  or  metric  defined  as  a  function  of  elapsed 
calendar  time  r, . 

u;  (t,-)  jth  control  limit  segment  value  for  a  particular  control  limit  as  a  function 
of  elapsed  calendar  time  r, . 


Performance-Based  Forecasting  and  Re-Baselining 

At  the  heart  of  the  PPMC  vision  is  the  desire  to  forecast  the  final  project  outcome.  We 
have  already  proposed  the  idea  of  including  the  estimation  process  and  all  of  its 
algorithmic  strength  in  the  forecasting  process.  The  question  now  becomes,  “How  can 
we  do  that?”  The  following  subordinate  sections  contain  a  set  of  process  steps  that 
support  the  creation  of  a  new  project  plan  (estimate)  that,  by  its  nature,  represents  a 
performance-based  forecast  and,  therefore,  represents  a  reasonable  new  baseline  plan 
(re-baseline).  A  strategic  goal  of  PPMC  is  to  eventually  automate  as  many  steps  in  this 
process  as  are  reasonably  possible. 
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Step  1 :  Start  a  New  Estimate 

Use  the  current  baseline  as  the  starting  point  for  a  new  estimate. 


Step  2:  Update  Size  Estimate 

Revisit  all  of  the  assumptions  (Basis  of  Estimate)  associated  with  the  current  estimate  of 
and  uncertainty  around  software  size.  Include  other  sizing  techniques,  if  appropriate,  in 
order  to  improve  sizing  accuracy.  Make  changes  where  appropriate. 


Step  3:  Update  Technology  Assumptions 

Revisit  all  Knowledge  Base  and  technology  parameter  assumptions  that  formed  the 
Basis  of  Estimate  for  the  current  baseline.  Align  Knowledge  Base  selections  and 
technology  parameter  settings  with  what  has  actually  happened  to  date  on  the  project. 
Make  changes  where  appropriate. 


Step  4:  Update  Schedule  Assumptions 

Revisit  the  project  start  date  and  any  assumptions  about  the  time-effort  tradeoff  solution 
(i.e.,  minimum  time  versus  optimal  effort,  schedule  constraints,  etc.).  Make  any  changes 
where  appropriate. 


Step  5:  Update  Staffing  Assumptions 

Specify  staffing  constraints  from  project  start  to  time  now  that  match  (as  close  as 
possible)  the  actual  staffing  profile  to  date.  Revisit  future  staffing  assumptions  from  time 
now  and  make  adjustments  where  appropriate. 


Step  6:  Update  Labor  Rate  and  FTE  Assumptions 

Revisit  all  labor  rates  as  well  as  the  definition  of  a  Full  Time  Equivalent  person  (typically 
specified  as  number  of  labor-hours  per  labor-month).  Make  any  changes  where 
appropriate. 


Step  7:  Time  Now  Calibration 

This  step  assumes  that  the  estimation  model  or  process  being  used  can  be  calibrated 
by  making  adjustments  to  one  or  more  calibration  coefficients.  We  propose  iteratively 
adjusting  each  calibration  coefficient  with  the  goal  of  creating  a  plan  that  corresponds 
with  actual  performance  (BCWP)  and  actual  expenditure  (ACWP)  to  date.  We  propose 
a  reasonable  test  for  this  correspondence  to  be  performance  indices  (BPI,  CPI,  SPI, 
TPI)  approaching  unity.  Practically  speaking,  we  are  creating  a  new  plan  that  matches 
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what  has  already  happened  and  uses  established  estimation  relationships  to  predict 
what  is  likely  to  happen  from  time  now  considering  what  has  already  happened  up  to 
time  now. 


Step  8:  Communicate  the  Results 

Use  an  appropriate  set  of  charts  and  reports  to  present  the  results  of  the  forecasting 
process.  A  chart  that  is  particularly  useful  for  this  purpose  is  of  the  type  shown  in  Figure 
8. 

Step  9:  Re-Baseline  the  Project 

If  warranted  by  project  expectations,  goals,  and  commitments  and  upon  approval  by 
appropriate  authority,  replace  the  current  baseline  with  the  current  (new)  estimate  and 
track  against  this  new  baseline. 

Communicating  Performance  Measurement  Information 

PPMC  Outputs 

One  of  the  primary  goals  of  PPMC  is  to  provide  adequate  supporting  documentation 
(charts  and  reports)  to  support  the  software  project  management  process  and  to  satisfy 
stakeholder  needs.  Figure  8  through  Figure  14  show  prototypical  examples  of  charts 
and  reports  that  can  be  used  for  such  a  purpose. 
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Figure  8:  Prototype  Performance  Measures  Tracking  Chart 


Figure  9:  Prototype  Performance  Indices  Tracking  Chart 
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1  ^  Health  &  Status  Indicator 
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Figure  10:  Prototype  Status  Indicators  Chart 
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Figure  1 1 :  Prototype  Schedule  Plan  versus  Prediction  Chart 
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Figure  12:  Prototype  Effort  Plan  versus  Prediction  Chart 
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Figure  13:  Prototype  Defects  Tracking  Chart 
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Figure  14:  Prototype  PPMC  Report 


Summary  and  Conclusion 


Purpose  Revisited 

This  paper  has  reviewed  the  fundamentals  of  software  project  management  and  of 
Performance  Measurement  (including  proposed  extensions)  and  then  examined  a 
proposed  process  called  Parametric  Project  Monitoring  and  Control  (PPMC)  whereby 
accepted  algorithms  currently  used  for  software  cost  and  schedule  estimation  during  the 
project  planning  process  are  incorporated  into  the  forecasting  and  re-baselining 
processes  to  yield  a  more-realistic  time-range  prediction  of  a  project’s  cost  and  duration. 

Areas  of  Future  Study 

The  following  are  suggested  opportunities  for  improving  PPMC: 

Identify  new  measures  and  metrics  that  can  provide  better  insight  into  project  health  and 
status. 

Investigate  methods  for  automating  the  Performance-Based  Forecasting  and  Re- 
Baselining  Process.  Regarding  the  iterative  calibration  process,  it  might  be  possible  to 
develop  a  system  of  equations  that  combine  the  fundamental  software  estimation 
equations  with  the  equations  for  calculating  Performance  Measurement’s  performance 
indices  such  that  the  calibration  coefficients  could  be  resolved  directly;  i.e.,  avoiding  the 
iterative  guessing  process. 

Develop  new  graphic  and  tabular  representations  of  the  measures  and  metrics  data  that 
improve  communicating  a  project’s  health  and  status. 

Investigate  the  idea  of  some  sort  of  project  diagnostic  expert  system  that  would 
combine  PPMC  measures  and  metrics  and  provide  a  list  of  suggested  potential  project 
problems  that  are  indicated  by  the  given  values  of  these  measures  and  metrics. 
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The  Road  to  Process  Improvement  Successes: 

CMMI  Level  5/ISO  9001 :2000  Business  Model 
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BAE  Systems  Lines  of  Business 
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BAE  SYSTEMS 


National  Security  Solutions 


Remarkable  People 
Remarkable  Performance 
Trusted  Partner 

BAE  SYSTEMS 
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National  Security  Solutions 
Employees  and  sites 


BAE  SYSTEMS 


-56  sites  across  the  US 
-Nine  sites  in  other  countries 


Dmahi 


SL  Louis.  MO 


Arnold.  MO 


Rome.  MY 


Pittsburgh  r  PA 
Dayton.  OH 


Burlington 4  MA 


Tampa.  FL 


Piano,  IX 


Hanover,  NH 


Ogden,  UT 

Mountain  View,  CAi 
China  Lake.  CA 

San  Diego.  CA 


Longmont,  CO 

Denver,  CO 
Colorado  Springs.  CO 


FT  Worth.  TX 


ML  Laurel  r  NJ 

Geihesda.  MD 
Gaithersburg,  MD 
Washington,  DC 
Arlington,  VA 
New  region,  VA 
Dahlgren,  VA 
Reston,  VA 
Hampton,  VA 
Norfolk,  VA 
Chantilly,  VA 
Warner  Robins.  GA 


Other  Locations  Include 


Hawaii 
Australia 
United  Kingdom 


CS-4 

10/24/2005 


National  Security  Solutions 
Business  Areas 


BAE  SYSTEMS 


Systems  Integration 

Large-scale  system-of-systems  integration  of  information  systems  for  the  defense 
and  intelligence  community 


Intelligence  Systems 

Image  management  and  exploitation  systems  for  mapping,  charting,  geodesy,  and 
intelligence  applications 


Defense  Systems 

Image  processing,  mission  management,  and  C4ISR  technologies  for  end-to-end 
mission  performance,  targeting  and  test  solutions  for  advanced  defense  electronic 
systems 


Geospatial  Products  and  Solutions 

Commercial  Software  for  Photogrammetry,  Mapping  &  GIS,  Imagery  Exploitation, 
C4ISR,  Targeting,  Visualization  &  Simulation,  Natural  Resource  Management,  and 
Vertical  Obstruction  Identification 


Advanced  Information  Technologies 

Developing  advanced  technology  solutions  that  provide  integrated,  high  performance 
capabilities  for  the  entire  information  chain. 
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Road  to  Process  Improvement  (PI)  Successes 

CMMI  Level  5  /  ISO  9001 :2000  Business  Model 

Topics  Covered 

-PI  Evolution  Roadmap 
-PI  Expansion 

-PI  and  Business  Impacts/Value  Realized 
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PI  Evolution  Roadmap 

NSS  Process  Improvement  History:  CMM/CMMI 
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Process  Improvement  Achievements 
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PI  Evolution  Roadmap 

Process  Improvement  Organization  Elevated  and  Broadened 
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■  Elevated  visibility 

■  Managed  as  a  project 

■  Phase  reviews 

■  Separate  budget,  resources, 
schedule 

■  Metrics 

■  Periodic  reviews  with  Exec  Mgt 
(monthly) 
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PI  Evolution  Roadmap 

NSS  Process  Improvement  History:  CMM/CMMI  and  ISO 
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Process  Improvement  Achievements 
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PI  Evolution  Roadmap 

ISO  9001  -  Quality  Management  System  Evolution 

-  Prior  to  September  1994 

-  Approved  to  Military  Standards 

-  Changing  DoD  initiatives  for  procurement  reform 


BAE  SYSTEMS 


-  Issues 

-  Given  12  months  to  achieve  initial  ISO  9001  certification  (typically  took  18  months, 
and  70%  failed  assessment  the  first  time) 

-  Documentation  overkill  and  no  time  to  re-architect 

-  Lack  of  engineering-specific  process  documentation 

-  Poor  control  and  management  of  records  and  training  needs 


-  Improvements 

-  Selected  personnel  to  receive  formal  ISO  Implementation  and  Auditor  Training 

-  Established  multi-disciplined  ISO  Steering  Committee 

-  Conducted  company-wide  ISO  Awareness  Training 

-  Generated  requirements-to-document  trace  mechanism 

-  Developed  and  implemented  required  process  documentation  with  increased  focus 
on  engineering  practices 

-  Strengthened  the  commitment  to  training 

-  Implemented  company-wide  mechanism  for  control  of  records 

-  Achieved  ISO  Certification  in  8  months  and  passed  the  first  time 
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PI  Expansion 
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PI  Expansion 

Entire  Management  System 


Management 
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(MSG)  provides 
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PI  Expansion 

Membership 
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PI  Expansion  | 

Entire  Management  System  -  Shared  Tools/Mechanisms/Enablers 
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-Process  Change  Request  (PCR) 

-Document  Restructure  Team  (DRT) 

-Sector-Wide  Integration  of  Requirement  Mapping  (SWIRM) 
-Corrective/Preventive  Action 
-Quality  Audit  System  (QAS)  and  Process  Health 
-Customer  Satisfaction 
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PI  Expansion 

Process  Change  Request  (PCR) 


BAE  SYSTEMS 


-  Employees  provide  feedback  (changes  and  additions)  on  processes,  tools,  and  training 
material 

-  Web  based  PCR  form  implemented 

-  Easy  use  resulted  in  increased  visibility  and  involvement 

-  Document  PCRB  decisions 

-  PCRs  assigned  to  process  owners  for  evaluation  and  implementation 


* Process i 


Process  Change  Request 

Hint:  Place  cursor  over  red  tent  to  reveal  hints 


PCR  Homs 
PCR  Users  Guide 


>  ideas  ■ 


From; 

Iwt  name 

first  name 

Hinall  nililrmw  |klione  number 

1  l 

Source: 

l 

Criticality: 

1  l 

i.JSylj  a  mi.  oa  m 

Critical  lly  Hnwrliitiou: 

|  Individual 

1 Lcw  zl 

i 

Topic: 

KuqaidiiHj: 

[Choose from  the  list 

l\  Subject:  ]  Choose  from  ihe  list 

Auq  until: 

*1 

"3 

Submit  PCR  | 

A 

N ot| lv  Ail iJHI o  n  o  1  P ii POO nw 
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PI  Expansion 

Process  Document  Structure 


BAE  SYSTEMS 


CN  -  Method  to  communicate  an  interim  change  to  an  existing  policy,  process  or 
procedure  or  a  new  policy,  process  or  procedure  that  has  not  been  released  to  DOLLS. 

PL  -  High  level  policy  statement  defining  the  Functional  and/or  Organizational  conduct 
of  operations  and  assigning  authority  and  responsibility  for  implementation. 

PS  -  Summary  descriptions  of  a  process.  May  include  diagrams  of  process 
flows  where  it  adds  value  to  understanding  the  process. 

PD  -  Detailed  step-by-step  procedural  instructions  of  what  precisely  has  to  be 
done  to  achieve  a  specific  output. 

FM  -  Form  that  cannot  be  altered  that  identifies  specific  information  for 
insertion  and  for  which  completion  results  require  retention. 

TM  -  Template  that  can  be  used  by  a  project  as  a  starting  point. 

CK  -  Checklist  or  other  Aide  to  complement  a  Process 
or  a  Procedure  document. 

RF  -  Reference  Doc  to  complement  a  Process  or 
a  Procedure  document. 

TR  -  Types  of  training  material  to 
include:  Formal  course  material 
developed  by  the  company,  job  aides, 
conference  /  brown  bag  presentations, 
etc. 
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PI  Expansion 

Sample  Branch  of  the  Integrated  Engineering  Process  Document  Tree 


BAE  SYSTEMS 


Engineering 
Development 
Policy  (PL) 


Perform  System 
Acceptance 
Test  Process  (PS) 


Test  Design 
Template  FM 


N  S' 


y  v 


t 


Test  Plan 
Template  FM 


Perform  System 
Acceptance  Test- 
"est  Design  Procedure  (PD) 


I  I  I  I  I 


y  \ 


T 


Test  Description 
Example  RF 


GFE/GFI 
Template  RF 


/  V 


T 


Component-Based 
Testing  White  Paper  RF 


Component  Test 
Plan  Template  RF 


FM  =  Form 
RF  =  Reference 
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PI  Expansion 

Process  Document  Structure 


BAE  SYSTEMS 


Company-wide  Directives  Restructure 

Established  a  Document  System  that  Fosters  and  Promotes  Continuous  Improvement 


Home  Grown  System 

❖  Distributed  Environment 

>  Many  uncontrolled 
copies 

Special  formats 
stored  for  multiple 
users  in  multiple 
places 

Publishing 
Responsibility 

>  Hardcopy  Signatures 

From 


Document 

On-Line 

Lookup 

System 

(DOLLS) 


COTS  Implementation 
❖  Central  Repository 

>  Single  master  with  one  native 
format  stored 

>  Common  look  and  feel 

>  Standardization  across  the 
business 

>  Greatly  improved  locating 
proper  operating  document 

>  Easily  transportable  to  others 

>  Electronic  Approval 

To 
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PI  Expansion 

Sector-Wide  Integration  of  Requirements  Mapping  (SWIRM) 


BAE  SYSTEMS 


Maintain  Document  Compliance  and  Trace 


> 


*  Relevant  Stakeholders:  **  Specification  (Spec)  Requirements: 

•Process/Document  Owners  -CMMI 

•Compliance  Authority/Coordinators  *ISO  9001 

•Corporate/Management  Directives 
•Contractual/Regulatory  Requirements 
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PI  Expansion 

Quality  Audit  System  (QAS)  and  Process  Health 


BAE  SYSTEMS 


Goal:  Exceed  90%  health. 

Goal  Status:  All  Process  Areas  exceeded  90%. 

PROCESS  HEALTH 

PROGRAM  LEVEL  AUDITS  ~  BY  PROCESS  AREA 


12  MONTH  PROCESS  HEALTH  3  MONTH  PROCESS  HEALTH 

'01-JUN-04  thru  '31-MAY-05  '01-MAR-05  thru  '31-MAY-05 


PROCESS  AREA 

OBSERVATIONS 
High  |  Med  |  Low 

DISCREPANCIES 
High  |  Med  |  Low 

Health 

OBSERVATIONS 
High  |  Med  |  Low 

DISCREPANCIES 
High  |  Med  |  Low 

Health 

KEY  PRACTICE  -  QUICK  CHECK 

22 

629 

0 

2 

22 

0 

96.1% 

12 

314 

0 

2 

16 

0 

94.1% 

PROJECT  PLANNING 

59 

83 

46 

1 

0 

0 

99.1% 

13 

24 

14 

0 

0 

0 

100.0% 

PROJECT  MONITORING  AND  CONTROL 

61 

199 

25 

2 

0 

4 

98.4% 

0 

0 

0 

0 

0 

0 

n/a 

SUPPLIER  AGREEMENT  MANAGEMENT 

21 

25 

9 

0 

0 

0 

100.0% 

9 

15 

7 

0 

0 

0 

100.0% 

INTEGRATED  PROJECT  MANAGEMENT 

52 

147 

66 

0 

0 

2 

99.8% 

0 

0 

0 

0 

0 

0 

n/a 

RISK  MANAGEMENT 

68 

99 

0 

0 

1 

0 

99.6% 

13 

26 

0 

0 

0 

0 

100.0% 

QUANTITATIVE  PROJECT  MANAGEMENT 

19 

87 

67 

0 

4 

0 

97.3% 

5 

21 

14 

0 

0 

0 

100.0% 

REQUIREMENTS  MANAGEMENT 

93 

45 

13 

0 

0 

2 

99.7% 

24 

10 

2 

0 

0 

0 

100.0% 

REQUIREMENTS  DEVELOPMENT 

91 

105 

51 

0 

0 

0 

100.0% 

0 

8 

0 

0 

0 

0 

100.0% 

TECHNICAL  SOLUTION 

138 

213 

54 

0 

3 

1 

99.3% 

16 

12 

0 

0 

0 

0 

100.0% 

PRODUCT  INTEGRATION 

143 

140 

185 

0 

1 

8 

99.2% 

26 

32 

27 

0 

0 

0 

100.0% 

VERIFICATION 

230 

180 

179 

1 

0 

0 

99.7% 

50 

33 

39 

0 

0 

0 

100.0% 

VALIDATION 

95 

14 

110 

1 

0 

2 

98.9% 

11 

0 

12 

0 

0 

0 

100.0% 

CONFIGURATION  MANAGEMENT 

40 

54 

23 

1 

1 

0 

97.9% 

18 

17 

11 

1 

0 

0 

96.5% 

PROCESS  AND  PRODUCT  QUALITY  ASSURANC 

182 

126 

40 

7 

3 

2 

96.5% 

0 

0 

0 

0 

0 

0 

n/a 

MEASUREMENT  AND  ANALYSIS 

84 

110 

34 

2 

0 

0 

98.6% 

11 

12 

5 

1 

0 

0 

94.4% 

DECISION  ANALYSIS  AND  RESOLUTION 

34 

22 

29 

0 

0 

0 

100.0% 

0 

0 

0 

0 

0 

0 

n/a 

CAUSAL  ANALYSIS  AND  RESOLUTION 

18 

14 

18 

0 

0 

0 

100.0% 

3 

10 

15 

0 

0 

0 

100.0% 

GENERIC  PRACTICES 

12 

31 

39 

0 

0 

0 

100.0% 

0 

0 

0 

0 

0 

0 

n/a 

TOTAL 

1462 

2323 

988 

17 

35 

21 

98.6% 

211 

534 

146 

4 

16 

0 

97.6% 

Data  Source:  Quality  Audit  System.  Includes  completed  audits  performed  during  time  period  indicated  only.  A 
QAS  Query  Date  '01-JUN-05  completed  audit  is  one  with  all  audit  questions  completed  and  the  date  the  last  question  was  audited  occured  during 

the  indicated  time  period. 
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PI  Expansion 

Corrective  Action  (C/A)  and  Preventative  Action  (P/A) 


BAE  SYSTEMS 


C/A 


Root  Cause 

Action  to  Prevent  Recurrence 
(Prevent  Recurrence) 

I  P/A  I 


Determine  cause  and  take 
actions  to  mitigate/eliminate 
potential  discrepancies. 
(Prevent  Occurrence) 
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PI  Expansion 

Corrective/Preventive  Action  Reporting/Tracking  Systems 


BAE  SYSTEMS 


Product 

Reviews 


Supplier 
Performance 
Index 


Design 
Reviews 


Quality  Audit  System  (QAS) 

Priority  Corrective  Action  Request  (PCAR) 
Quality  Evaluation  Report  (QER) 

Defect  Prevention  Causal  Analysis  Reports 
Return-To-Green  Process  Template 
Action  Item  Database 
Engineering  Corrective  Action  Tracking 
Risk  Registry  Database 
Lessons  Learned  Database 


Inspection 
Checklists/Forms 


Risk 
Management 
Plans 


Process  Audits 


GFP 

Lost/Damage 
Reports 


Discrepancy 
Reports 
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PI  Expansion 

Internal  Customer  Satisfaction 


BAE  SYSTEMS 


Information  Technology 
Customer  Satisfaction  Survey: 
Common  Company  Services 


Q1 

Q2 

Q3 

Q4 

Q5 

Q6 

Q7 

Q8 


TJ 


0% 


20% 


40% 


60% 


80% 


100% 


■  Excellent 

□  Satisfactory 

□  Marginal 

■  Unsatisfactory 


Q1  Centralized  reprographics  services. 

Q2  Phone  services  (telephone,  voicemail,  teleconferencing,  pagers). 

Q3  Electronic  presentation  room  (EPR)  and  video  teleconference  (VTC)  services. 

Q4  On-site  desktop  computer  support  (installs,  moves,  adds,  changes). 

Q5  Corporate  application  services  (Time  Reporting,  Employee  Self  Service,  Travel) 
Q6  How  well  the  IT  investments  supported  the  business  goals  you  defined  this  year. 
Q7  IT  extended  services  (computer  sales/purchase/reimbursement  program). 

Q8  Access  to  Internet  and  company  facilities. 


Overall  Survey  Stats 

1473  survey  requests 
320  respondents 
(Return  Rate:  22%) 
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PI  Expansion 

External  Customer  Satisfaction 


BAE  SYSTEMS 


RATING  BY  CATEGORY  | 

PROG 

CUSTOMER/  PROGRAM 

PROD 

PERF 

SYS 

ENG 

SOFT 

ENG 

LOGIST 

SUPPRT 

PROD 

ASSUR 

SCH 

COST 

CONTRL 

MGMT 

RESP 

SUBCN 

MGMT 

PROG 

MGMT 

COMM 

12  mo 

run  avg 

DEFENSE  SYSTEMS 

i 

S 

XXX 

G 

G 

P 

B 

G 

P 

G 

P 

G 

P 

B 

3.8 

S 

XXXXXXXX 

B 

B 

N/A 

N/A 

N/A 

B 

B 

B 
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B 

G 
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S 
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B 
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B 

B 

B 

P 

B 

B 

B 

P 
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S 

xxxxxxxxxxx 

B 

B 

B 

B 

B 

B 
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B 
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B 

B 

4.9 

S 

xxxxxxxxxxxxxx 

B 

B 

B 

P 

B 

B 

B 

B 

N/A 

B 

B 

4.8 

S+ 

xxxxxxxxxx 

G 

P 

P 

P 

G 

P 

B 

P 

G 

P 

P 

3.8 

S+ 

xxxxxxxxxxxx 

P 

P 

P 

G 

P 

B 

P 

B 

G 

B 

P 

4.2 

s 

xxxxxxxxxxxxxxxxx 

B 

P 

B 

B 

B 

B 

P 

B 

N/A 

B 

B 

4.6 

s 

xxxxx 

P 

N/A 

N/A 

N/A 

G 

G 

P 

B 

B 

P 

P 

4.0 

c 

XXXXXX 

B 

B 

B 

P 

B 

B 

B 

B 

N/A 

B 

B 

5.0 

s 

XXXXXX 

B 

B 

B 

B 

B 

B 

B 

B 

N/A 

B 

B 

5.0 

c 

xxxxx 

B 

B 

P 

N/A 

B 

B 

B 

B 

N/A 

B 

B 

4.6 

s 

XXXXXXXX 

B 

B 

B 

B 

B 

B 

B 

B 

N/A 

B 

B 

4.9 

c 

xxxxxxxxxxxxxxx 

B 

P 

P 

P 

G 

P 

P 

B 

N/A 

B 

B 

4.3 

c 

xxxxx 

B 

G 

G 

G 

G 

G 

B 

G 

N/A 

B 

G 

3.6 

s 

XXX 

N/A 

B 

B 

N/A 

P 

P 

P 

P 

N/A 

P 

P 

4.2 

s 

xxxx 

G 

P 

N/A 

N/A 

G 

G 

N/A 

G 

N/A 

G 

G 

3.0 

s 

xxxx 

P 

P 

P 

B 

P 

P 

G 

P 

P 

P 

P 

4.2 

c 

XXX 

G 

P 

P 

P 

G 

B 

P 

B 

N/A 

B 

B 

4.1 

c 

XXX 

P 

P 

P 

N/A 

N/A 

P 

N/A 

B 

N/A 

B 

P 

4.0 

c 

XXX 

P 

B 

B 

N/A 

B 

B 

P 

B 

N/A 

B 

B 

4.1 

c 

xxxxxxxxxxxxxx 

P 

P 

P 

P 

G 

B 

P 

B 

N/A 

B 

B 

4.2 

c 

xxxxx 

G 

P 

G 

N/A 

G 

N/A 

P 

B 

N/A 

B 

B 

4.0 

INTELLIGENCE  SYSTEMS 

C 

XXXXXX 

N/A 

P 

B 

N/A 

P 

B 

P 

P 

N/A 

P 

P 

4.3 

S 

xxxxxxx 

G 

P 

B 

G 

G 
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P 

N/A 

P 

G 

3.8 
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P 

P 

P 
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P 
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N/A 

P 

P 

4.1 

C 
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P 
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P 

B 

B 

N/A 

P 

P 

4.3 

C 
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N/A 

G 

G 

N/A 

G 

P 

G 

G 

N/A 

G 

G 

3.1 

C 

xxxx 

P 

B 

P 

B 

P 

P 

P 

B 

B 

B 

B 

4.5 

C 

xxxxx 

G 

P 

P 

G 

N/A 

G 

G 

P 

N/A 

P 

P 

3.7 

1  Process  Summary  -  Customer  Rating  | 

B 

Percentage 

33.3% 

35.3% 

31.0% 

36.7% 

28.5% 

46.1% 

37.8% 

68.0% 

41.6% 

55.5% 

57.5% 

P 

12  Month 

40.7% 

50.8%  ! 

57.1% 

48.6% 

36.8% 

35.8% 

49.7% 

24.6% 

10.1% 

40.3% 

31.8% 

G 

Running 

25.9% 

13.9%  3 

1 1 .9% 

14.7% 

34.7%  1 

18.1% 

12.5% 

i  7.5% 

48.3% 

4.1% 

10.5% 

Y 

Average 

R 

|  Overall  Average  | 

1  ^  1 
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PI  Expansion 

Transferring  Best  Practices 


BAE  SYSTEMS 


Collaborated  Architecture  within  the  Organization 

Shared  proven  Organizational  Standard  Set  of  Processes  Across 
Enterprise 


National  Securit 
Solutions 


Engineering 

Process 

Program 

Quality 

Improvement 

Management 

Assurance 

*  Business 
;  Development 

Business 

Operations 

Contracts 
&  Legal 

Human 

Resources 

Information 

Technology 

— 

Adopted 

By: 

Platform 

Solutions 

Shared 

With: 

Technology  Solutions,  Flight  Systems,  IDS-Austin, 

CS&S  UK,  AeL  UK,  Australia 

Advance  Integrated  Technologies,  Land  &  Armament, 
and  others  ... 

1 0/24/2005 


PI  Expansion 

Operating  Group  (OG)  Regulatory  Panels 


BAE  SYSTEMS 


-  Regulatory  Panels 

-ISO  9001  and  AS  9100 


-  ISO  14001  Environmental 

-  CMMI 

-  Representative  from  each  Line  of  Business 

-  Purpose: 

-  Remove  walls  that  may  have  existed  between  sites 

-  Benchmark  and  share  best  practices  within  each  site 
and  leverage  on  them 

-  Harmonize  and  integrate  regulatory  requirements 

-  Establish  and  implement  OG  level  policies  to  operate  as 
a  borderless  enterprise 
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PI  and  Business  Impacts/Value  Realized 

Improvements  Over  Past  10  Years 


BAE  SYSTEMS 


Project  Measure 

Then 

Now 

Actual  vs.  Negotiated  cost 

+/-  40% 

+/-  7% 

Cost  Performance  Index 

unknown 

1.03 

Schedule  Performance  Index 

unknown 

0.99 

Average  Award  Fee 

90% 

98.10% 

Greens  on  Customer  Sat.  Survey 

90% 

1 00% 

Process  Measure 


Capability  Maturity  Model  Int.  (CMMI) 


5 
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PI  and  Business  Impacts/Value  Realized 

Improved  Measurement  Reliability 


BAE  SYSTEMS 


Monthly 

Metrics  Verification  and  Validation  Compliance 
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PI  and  Business  Impacts/Value  Realized 

Less  Variability  &  Increased  Predictability 


BAE  SYSTEMS 


Average  Calendar  Days  to  Resolve  DRs  (DR  Cycle  Time) 


1000.0 
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500.0 


250.0 
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Q1  Q2  Q3  Q4  Q1  Q2  Q3  Q4  Q1  Q2  Q3  Q4  Q1  Q2  Q3  Q4  Q1 

Period  of  Performance 
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PI  and  Business  Impacts/Value  Realized 

Defect  Reduction 


BAE  SYSTEMS 


Post  Delivery  Defect  Density  Reduction  Goal  (2002-2005) 
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Recorded  Quarter  Defect  Density 


— ■ —  Post-Delivery  Defect  Density  Reduction  Goal 
-  -4-Quarter  Moving  Average 
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PI  and  Business  Impacts/Value  Realized 

Project  Example  -  Defect  Reduction 


BAE  SYSTEMS 


O 

o 

l 

tfS 

$ 

0) 

Q 


Project  Example 

12-Month  Post  Delivery  Defects/KSLOC 
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PI  and  Business  Impacts/Value  Realized 


BAE  SYSTEMS 


-  Functionally 

-  Integrated  end-to-end  project  procedures 

-  Aligned  Business  Drivers  and  Program  Supports  more  closely 

-  Established  a  process  framework  for  internal  projects  throughout  the  organization 

-  Improved  internal  customer  relationships  and  teamwork 

-  Organizationally 

-  Established  a  common  architecture  framework 

-  Identified  commonalities  while  maintaining  unique  Business  model  supports 

-  Provided  the  opportunity  for  Organizational  Business  Areas  to  work  together  more  effectively  on 
common  projects 

-  Reduced  Customer  oversight  in  daily  and  annual  review,  audit,  and  product  inspection  activities 

-  Reduced  time  spent  preparing  for  audits  due  to  customer  audit  avoidance  and  reduction  in  audit 
scope 

-  Increased  our  competitive  edge  and  market  opportunities 

-  Across  the  Enterprise 

-  Shared  best  practices  implementations  and  lessons  learned 

-  Adopted  a  mature/proven  Organizational  Standard  Set  of  Processes  (OSSP)  as  the  basis  for 
continuous  process  improvement 

-  Broadened  the  scope  of  continuous  improvement  communications 
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Summary 


BAE  SYSTEMS 


-  Learned  from  failures  to  emerge  with  an  even  stronger  focus  on 
continuous  improvement 

Elevated  process  improvement  importance/focus  across 
organization/enterprise 

-  Recognized  the  need  for  a  unified  system  for  process  improvement 
and  quality  management  using  CMMI  for  depth;  using  ISO  for  breadth 

Established  an  Organizational  Process  Group  -  Expansion  (OPGE) 
that  involves  all  Business  Areas  and  OG  relevant  stakeholders 

Established  ISO/CMMI  regulatory  panels  at  the  OG  to  maintain  the 
focus  on  continuous  improvement 

-  Recognized  both  the  qualitative  and  quantitative  value  leading  to 
more  efficient  functional  and  project  performance 

-  Improved  our  competitive  edge  and  increased  value  to  our  customers 
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The  Road  to  Process  Improvement  Successes: 

CMMI  Level  5/ISO  9001 :2000  Business  Model 


Contacts 


Name: 

Vicki  Moore 

Tel: 

(858)  592-5715 

email: 

vicki.moore@baesvstems.com 

Name: 

Rosetta  Phelps 

Tel: 

(858)  675-1715 

email: 

rosetta.Dhelps@baesvstems.com 

Name: 

Debra  Roy 

Tel: 

(858)  592-5821 

email: 

debra.rov@baesvstems.com 
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NORTHROP  GRUMMAN 


A  Project's 

)3ce  Perspective  of 

CMMI  Level  5 


5th  Annual  NDIA  CMMI  Technology  Conference 


November  14-17,  2005 

Warren  Scheinin 
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Agenda 


Background 


■  In  2003,  the  Systems  Development  Operation 
organization  was  assessed  at  CMMI  Level  5  in  a 
externally-led  SCAMPI-ARC  A  appraisal 

■  This  organization  is  currently  preparing  for  a 
re-appraisal  next  month 

■  This  presentation  examines  some  of  the  lessons 
learned  and  benefits  associated  with  that  journey 

■  New  projects  cannot  rest  on  the  laurels  of  past 
projects  but  must  proactively  plan  for  activities  at  all 
levels  of  the  CMMI  model 

■  It  takes  time  to  record  what’s  going  on,  but  the 
resulting  evidence  is  invaluable  to  the  project 
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Climbing  the  CMMI  Level  5  Ladder 


Each  CMMI  Level  is  a  step  to  Project  Maturity 


Starts  with  the  foundations  for  a 
maintainable  system 


Level  5 
Optimizing 


Gets  your  head  above  water 

■  Clears  the  fog  of  fighting  fires 

■  Engage  the  supercharger 


The  Ad  Hoc  Sink  Hole 


Level  4 

Quantitatively 

Managed 


NORTHROP  GRUMMAN 
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Level  2:  Don't  Throw  Away  the 

Instruction  Manuals 


■  Know  what  it  is  you  promised  to  do 

■  Know  what  it  will  take  to  deliver  what  you  promised 

■  Know  what  others  promised  to  do 

■  Keep  track  of  expected  inputs 

■  Remind  suppliers  of  what  is  due 

■  Start  collecting  data  points 

■  Don’t  forget  the  past 

■  Configuration  Management  allows  reproduction  of 
deliverables  and  archives  management  decisions 

■  Ask  others  for  help 

■  Quality  Assurance  provides  a  check  on  progress 
and  credit  for  accomplishments 


NORTHROP  GRUMMAN 
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Level  3:  Combine  the  Islands  of  Excellence 

Into  a  Functioning  Team 


■  Engage  the  software  development  lifecycle 

■  Follow  the  instructions 

■  Be  able  to  prove  it  works  right  and  well 

■  Take  advantage  of  organizational  assets 

■  Not  invented  here  is  still  a  bad  idea 

■  Best  practices  will  save  time  and  money 

■  Stop  drowning  in  the  past 

■  Risk  management 

■  Peer  reviews 

■  Expand  beyond  your  borders 

■  Include  suppliers 

■  Include  Systems  Engineering 
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Level  4:  Understand  Your  Processes  and 

Subprocesses 


■  Co-ordinate  with  other  projects 

■  Take  advantage  of  organizational  knowledge 


■  Identify  the  implementation  of  processes  which 
perform  best 


■  Know  that  processes  are  performing  within  natural 
bounds  that  are  consistent  across  teams 


■  Six  Sigma 

■  Level  3  metrics, 
measurement  processes, 
and  goal  setting  are 
generally  inadequate  for 
Levels  4  and  5 

■  Need  better  definitions 
of  the  measures 

■  Lower  level  metrics  of 
subprocesses 
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Level  5:  Get  Ahead  of  the  Curve 


Catch  problems  before  they  attack  your  project 
Level  3  firmly  in  place 
Reduce  the  variation 

■  Train  people  on  the 
process 

■  Create 

procedures/checklists 


Increase  the  effectiveness 
(increase  the  mean) 

■  Train  people 

■  Create  checklists 

■  Reduce  waste  and  re-work 

■  Replicate  best  practices 


UCL=  11.17 


Mean=7.268 


LCL=3.363 


Revolutionary  Process  Capability 
Improvement 


NORTHROP  GRUMMAN 
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Why  is  Being  Appraised  so  Difficult? 


■  "The  telephone  is  the  greatest  single  enemy  of  scholarship;  for 
what  our  intellectual  forebears  used  to  inscribe  in  ink  now  goes 
once  over  a  wire  into  permanent  oblivion. " 


■  Finding  documented  evidence  for  a  CMMI  appraisal 
is  often  difficult  because  project  performers  often  do 
not  take  the  time  to  write  down  what  they  are  doing 

■  The  lack  of  written  records  sometimes  leads  to 
arguments  about  what  is  supposed  to  be  happening 

■  “Just  Do  It”  gets  the  job  done  in  the  short  term,  but 
written  records  are  necessary  to  reap  the  long  term 
benefits  of  operating  at  CMMI  Level  5 
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Undocumented  Results  Look  Great  But 
Fail  to  Reveal  Purpose  and  Process 


NORTHROP  GRUMMAN 
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Even  When  Documents  Are  Unearthed, 
They  Need  to  be  Understandable 


^  Benefits  are  There  (If  You  Know  Where  to  Look) 


GP  2.7  Stakeholder 
Involvement 


Cultural  dynamics  did  not  encourage 

communication  &  collaboration  across  project  organizations 


Permitted  “stove-piped”  responsibilities  within  software 


Project  oversight  not  independent 

^  Project  oversight  did  not  recognize  when  program  was  in  trouble 

Did  not  manage  ownership  by  each  employee 


Regressed  to  sell-off  criteria  vs.  delivering  a  working  system 


Fixing  bugs  took  precedence  over  system  stability 


Did  not  manage  involvement  of  end-users 


Continue  to  reinforce  Project  oversight  &  responsibility  per  new  policy 

■  ■  .  -  ■■  . 
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Benefits  Materialized  During  the  Climb 


■  Instituted  Weekly  CMMI  Coordination  Working  Group 

■  Collaborating  with  similar  projects  a  major  plus 

■  Task  list  and  schedule  showed  progress  and 
encouraged  participation 

■  Benefit:  Weekly  meetings  keep  the  momentum  going 

■  Took  full  advantage  of  upper  management  resources 

■  Monthly  S/W  Engineering  Process  Group  (SEPG) 
meetings  provided  moral  support,  training,  and  planning 

■  Benefit:  Presentations  by  Process  Assessment 
Organization  lead  clarified  principles  and  showed  top 
management  commitment 

■  Benefit:  Project  oversight  meetings  provided  conduit  for 
upper  management  help 

■  Benefit:  Evidence  book  reviews  by  top  managers 
assured  timeliness  and  quality  northror  Grumman 
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To  Be  Top-Tier  is  to  See  With  New  Eyes 


■  Process  improvement  is  built  into  the  system 

■  Evidence  Books  used  as  patterns  from  previous 
appraisals  were  not  sufficient  to  meet  later 
expectations 

■  Needed  to  add  more  evidence  as  our  understanding 
of  what  makes  a  good  process  has  grown 

■  The  culture  has  changed 

■  Process  improvement  is  the  object  of  many  CAR  and 
Six  Sigma  projects 

■  Process  people  are  not  the  first  to  go  when  budgets 
are  cut 

■  It  gets  easier  each  time 

■  Familiarity  leads  to  quicker  startup 

■  Less  training  needed,  less  resistance  to  change 

NORTHROP  GRUMMAN 
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Projects  Gain 


■  Produced  more  value-added  products  with  reduced 
effort  and  time 

■  Instead  of  overrunning  budgets  and  schedules, 
products  are  delivered  early  and  on  budget 

■  Needed  less  “help”  from  senior  management 

■  Lots  of  new  work  began  pouring  in 

■  Communications  with  other  groups  was  easier 

■  Meshed  well  with  cost  reduction  efforts 

■  Easier  to  understand  the  role  of  Systems 
Engineering  in  Software  Development 


NORTHROP  GRUMMAN 
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Project  Leaders  Gain 


■  More  up  front  thinking  means  less  work  later 

■  Fewer  problems  and  risks  along  the  way 

■  Improved  processes  added  slack  to  cost  and 
schedule  curves 

■  Fewer  replan  exercises 

■  Easier  to  give  back  resources 

■  Easier  to  help  other  projects 

■  Other  projects  consulted  us  to  find  out  why  things 
were  going  so  well 


NORTHROP  GRUMMAN 
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Individuals  Gain 


■  Better  understanding  of  how  to  get  job  done 

■  Less  stress 

■  Less  time  doing  rework 

■  Easier  to  transfer  from  project  to  project 

■  Easier  to  understand  need  of  Systems  Engineering 
in  Software  Development 

■  Concerns  were  escalated  more  quickly  to  the  proper 
level  of  attention 

■  More  enthusiastic  about  looking  for  improvement 
opportunities 

■  Down  side:  SPIN  meetings  are  much  less  popular 
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Initial  Resistance  to  Something  New  Faded 
Over  Time 


r 


*\  r 


Our  project  is 

Our  customer 

smaller  than 

doesn’t  care 

10  people. 

about  the 
CMMI. 

v _ “ _ ) 

f  \ 

k _ _ _ ) 

*\ 


We  didn’t  bid 
the  extra 
activities  in 
our  contract. 


Projects  must  comply  with  both  organizational  policies  and  contract 
requirements 

Even  if  your  customer  is  not  familiar  with  CMMI,  they  will  appreciate  the 
benefits:  CMMI  practices  the  customer 

Adopting  the  CMMI  is  a  cost  of  doing  business  and  is  included  in  the 
services  we  provide  our  customer  to  assure  quality  products 

Other  benefits 

-  Less  rework  ->  nights,  weekends  and  holidays  off 

-  Discussions  lead  to  “Ah  Ha’s”,  “I  thought...”,  “Oh,  I  didn’t  know...” 

-  No  surprises  -  the  customer  becomes  your  friend 

NORTHROP  GRUMMAN 
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Summary  —  Project's  Perspective  of  CMMI 
Level  5 


■  Much  of  the  hard  work  in  establishing  a  foundation 
is  past  with  significant  benefits 

■  Level  5  project  activities  put  available  information  to 
use  in  identifying  project  improvement  opportunities 

■  Innovative  process  improvements  are  readily 
available  for  implementation 

■  The  project,  management,  and  individuals  realize 
real  benefits  from  Level  5  operation 
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NORTHROP  GRUMMAN 


Defining  the  future 

Ways  to  Ensure 
the  Culture 
Supports  Level  5 


5th  Annual  NDIA  CMMI  Technology  Conference 

1 1m 


November  14-17,  2005 


Warren  Scheinin 

Systems  Engineer 
Northrop  Grumman  Corporation 


Agenda  —  Three  Components  of  a  Culture 
That  Supports  Projects  At  CM  MI  Level  5 


Vision:  Commitment  to  a  Strong 
Process  Infrastructure 
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Background 


■  In  2003,  the  Systems  Development  Operation 
organization  became  another  of  over  16  NGC 
organizations  assessed  at  CMMI  Level  5  in 
externally-led  SCAMPI-ARC  A  appraisals 

■  This  organization  is  currently  preparing  for  a 
re-appraisal  next  month 

■  This  presentation  examines  some  of  the  cultural 
components  that  made  this  journey  a  success 

■  Organizations  must  provide  vision  and 
encouragement  to  project  to  reach  CMMI  L5 

■  The  organizational  infrastructure  must  support  and 
guide  the  projects  to  CMMI  L5  performance 

■  Organizations  must  control  and  maintain  the 
forward  drive  and  infrastructure 
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Northrop  Grumman  Has  Evolved  A  Strong 
Process  Infrastructure 


■  The  NGC  Process  Infrastructure  is  built  on  the  corporate 
commitment  to  managing  programs  in  a  deterministic  way  to 
delivery  high  quality  products  at  lower  costs 

.  ■  New  programs  inherit  the  NGC  process  as  part  of  project 
startup 


Northrop  Grumman 
Software  Engineering  Processes 


Major  Contribution  to  MIL  and  DoD  Standards 


SDCE 

SCE 

SCAMPI 

2001 

l/S  12207 

SEiSW-CMM 

cmm 

(SW/SE) 

2  j-nmi 

ms 

ISOl/tEC  . . 

12207 

l  \ 

Derived  From 
Replaced  By 


"  * 

1906  VT  - 

2002  ^ 

IEEE  1498/ 

EIA  640  (Draft) 

WGMS  Optima  21™ 

(Single  Process  -  > 

Initiative) 

N QMS  Command 
Media  (PRM/SPM) 

1 

1997  T 

1994 

2000 

J-STD-M 

iSO 

ISO 

9001:1994 

9001:2000 

Project  Plans 
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NGC  Process  Structure  is  Integrated  with 
Teammates  and  the  Customer 


Certification 

Capability 

Performance 


Independent 

Audits 


Six% 

Profiling 

Certification 

Sourcing 

Contracting 

Managing 


O 

s 

$ 


Capability 

Risk 

Contract  Process 

Requirements 

Quality 

Performance 

Oversight 


Quality 

Performance 

Oversight 


Quantitative  Data 
Qualitative  Data 
Requirements  Flow 
Performance  Reports 
Opportunities 


Suppliers 


Assurance 


Quantitative  Data 
Qualitative  Data 
Requirements  Flow 
Performance  Reports 
Opportunities 

4 


Customer  Community 


N-Tier 


Risk  are  constantly 
identified  and  handled 


Risk 


Mission 


10/24/2005 1:56  PM 


HEADER  /  FOOTER  INFORMATION  (SUCH  AS  PRIVATE  /  CONFIDENTIAL) 


Copyright  2005  Northrop  Grumman  Corporation 


Organizational  Gains 


■  Organizational  and  Customer  goals  are  flowed  down 
to  project  goals. 

■  Stove  piles  are  replaced  with  continual  stakeholder 
involvement 

■  Concerns  are  escalated  more  quickly  to  the  proper 
level  of  attention 

■  Less  “help”  needed  by  projects  from  senior 
management 

■  Easier  to  transition  personnel  from  one  project  to 
another 

■  Less  firefighting,  more  emphasis  on  delivering 
consistently  high  quality  products  and  services  that 
meet  the  customer’s  needs 

■  Strong  program  performance  to  budget  and 

schedule;  lower  risk,  higher  quality  NOm ^  Cm/Ml 
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A  Culture  is  a  Way  of  Doing  Business 


■  Clear  Vision  instilled  in  goals  and  plans 

■  CMMI  and  Six  Sigma  practices  are  being 
institutionalized  as  the  normal  “way  of  doing 
business” 

■  Tied  to  strategic  goals  and  led  by  management 

■  Widespread  involvement  and  buy-in  at  all  levels 

■  Common  infrastructure  across  the  sector 

■  One  set  of  policies,  process  descriptions,  metrics, 
training 

■  Allows  for  common  enterprise-wide,  minimizes 
costs  and  promotes  best  practice  sharing  and 
adoption  across  the  organization 

■  Disciplined  configuration  management,  evaluation 
and  dynamic  improvements  for  increasing  capability 

NORTHROP  GRUMMAN 
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Mission  Success  Realization  System 

A  program  execution  system  based  on  proven  enterprise  policies  and  processes 


Customer  Requirements 


Policies,  Processes  &  Best  Practices 


i 

Program 

Deployed  System 

Execution 

r 


Process 

Corrective/ 

Improvement 

Preventive 

Action 

Six  Sigma 
Projects 


Reviews  / 
Audits 


Metrics  / 
Lessons 
Learned 


Customer 

and 

Program 

Feedback 


. . .  operating  in  a  closed-loop  system  aimed  at  Mission  Success 
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Characteristics  of  an  Effective  CMMI 
Culture 


■  Well-defined  Authority,  Roles,  and  Responsibilities 

■  Strong,  effective  infrastructure  with  closed-loop 
feedback  and  controls 

■  Flexibility  to  meet  needs  of  different  program  types 
and  degrees  of  application 

■  Independent  review  and  reporting  functions 

■  Well-qualified  workforce  able  to  manage  and  execute 
aspects  of  CMMI  at  all  levels  in  the  organization 

■  Evaluation,  appraisal,  improvement  and  configuration 
management  of  artifacts 

■  Commitment  to  performance  excellence  from  all 
levels  of  the  organization 

■  Commitment  to  customer  satisfaction 
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Authority,  Roles,  and  Responsibilities  Starts 
with  Senior  Management 


■  Organization  Leadership  sets  the  cultural  tone 

■  Senior  management  commitment  is  key  to  CMMI 
achievements 

■  Many  competing  initiatives 

■  Authority  defined  at  all  levels  of  organization 

■  Stakeholder  roles  and  responsibilities  defined 

■  Meetings  keep  the  momentum  going 

■  Need  to  keep  stakeholder  engaged 

■  Software/Systems  Engineering  Process  Group 

■  Communication 

■  Web  sites 

■  Emails  -  a  blessing  and  a  curse 

NORTHROP  GRUMMAN 
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Strong,  Comprehensive  Infrastructure 
Supports  the  Culture 


■  Organizational  Policies  and  Procedures  cover  all 
aspects  of  mission  success  and  engineering 

■  Training  provides  skills  and  acceptance 

■  Organization  standard  processes  allow  uniform 
project  processes 

■  Comprehensive  Measures/Metrics  system  provides 
estimates,  track  progress  and  allow  comparisons 

■  Process  Assets  Library  speeds  up  the  spread  of  the 
culture 

■  Lessons  Learned  database  prevents  errors 

■  Independent  SQA  reporting  path 

■  Senior  management  review 
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Web  Based  PAL  Gives  All  Performers  Easy 
Access  To  What  They  Need 


Process  Assets  Library 


NORTHROR  GRUMMAN 
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Organizational  Process  Capability  dB  Fuels 
the  Metrics  System  with  Vital  Feedback 


■  Organizational  Process  Performance 

■  Establishes  a  quantitative  understanding  of  the 
performance  of  the  organization’s  set  of  standard 
processes 


■  Provides  process  performance  data,  baselines,  and 
models  to  quantitatively  manage  the  organization’s 
projects 


Organizational 


project  performance 


organizational 
standard  process 


project’s 
defined  process 


organizational 
performance  data 
▲  &  models 


customer  and  project 
objectives 
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Web  Site  Makes  Best  Practices  and  Break 
Through  Improvement  Readily  Available 


Six  Sigma  Website  Startlt! 

Project  Database 

Six  Sigma  on-line  resources  keep 
employees  informed  and  knowledgeable. 


Green  Belt  Primer 


EE 

3 


ght  On  Six  Sigma 


Customers  in  Six  Sigma  —  live  Importance 
of  First  Impressions  ■= 


-*■1  •  £►*>-  *■  I*  ■►♦sir  ►  1  ►>  4 4*  V  i  — rtJ 


Monthly  Newsletter 
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Putting  it  All  Together:  Institutionalizing  the 
Improvements  in  a  Closed  Loop  System 


NORTHROP  GRUMMAN 
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Organizational 
Training  &  Tools 


Project  Plans 


Project  Results 


Project 

Schedules  &  Budgets 


Project 
Performance 


Industry/Government 

Standards 


Organizational 
Policies  &  Processes 


Metrics  Database 


■  Communications 

■  Sharing  best-practices 

■  Measurement  &  dashboards 


Mission  Systems'  Integrated  Approach 
to  Process  Improvement 


ISO  9001  i  I  CMMI 


r 


Business 

Measures 

Voice  of  the 
Customer 

1 

Change 

Management 

-L 

Methods 

DMAIC 

! 

Process 

&  Tools 

DFSS 

; 

Management 

ISO  9001  -  quality  management  discipline  for  project  and  functional 
areas 

Six  Sigma  -  framework  for  ensuring  process  improvements 
support  corporate  goals 

CMMI  -  use  of  industry  best  practices  in  software/systems 
engineering 

NORTHROP  GRUMMAN 
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Flow  Down  to  Projects 


■  Instituted  Weekly  CMMI  Coordination  Working  Group 

■  Collaborating  with  fellow  contracts  a  major  plus 

■  Benefit:  Weekly  meetings  keep  the  momentum  going 

■  Took  full  advantage  of  Division  resources 

■  Management  sponsorship  was  essential  to  success 

■  Monthly  SEPG  meetings  provided  support  group, 
training,  and  planning 

■  Benefit:  Presentations  by  Process  Assessment 
Organization  lead  clarified  principles  and  showed  top 
management  commitment 

■  Benefit:  PRA  meetings  conduit  for  upper  management 
help 

■  Benefit:  Evidence  book  reviews  by  top  managers 
assured  timeliness  and  quality 

NORTHROP  GRUMMAN 
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Institutionalization  Includes  Training  All 
Personnel 


Includes  how  to 
tailor 

Organizational 
Policies  & 
Processes  to 

^project  use. 


Module 

Test 

(Download  File 
Locally) 

Standard  Training  Module 

Revisl 

700  MISSION  SUCCESS  ! 

STM  700.0 

STM  700.0 

Policies  and  Processes  Awareness 

Rev  08,  0'j 

STM  700.1 

STM  700.1 

CMMI  Awareness 

Rev  02,01 

720  QUALITY  SYSTEM 

STM  723.0 

None 

Introduction  to  Documents  and  Records  Control 

Rev  08,  Oj 

STM  723.1 

None 

Documents  and  Records  Control 

Unavaij 

730  PROCESS  MANAGEMENT 

STM  731.0 

None 

Introduction  to  Organizational  Process  Definition 

Rev  07,  0} 

STM  731.1 

None 

Organizational  Process  Definition 

Rev  03,01 

None 

Introduction  to  Organizational  Process  Focus 

Rev  07,  oj 

r^STM  732.1 

None 

Organizational  Process  Focus 

Rev  03,  0'i 

STM  733.0 

None 

Introduction  to  Organizational  Process  Performance 

Rev  07,  0l 

STM  734.0 

None 

Introduction  to  Organizational  Innovation  and  Deployment 

Rev  07,  oj 

STM  735.0 

None 

Introduction  to  Organizational  Training 

Rev  08,  0'i 

STM  735.1 

None 

Organizational  Training 

Rev  03,  oi 

910  Acquisition  j 

sfffr^o 

None 

Introduction  to  Start-up  Planning 

Rev  05,  01 

920  Project  Management  ; 

STM  921.0 

^  STM  921.0 

Introduction  to  Project  Planning 

Rev  08,  01 

’  STM  921.1 

STM  921.1 

Tailoring 

Rev  07,01 

STM  921.1 

None 

Tailoring  Handout 

Rev  01 ,  oj 

STM  921.1 

None 

Tailoring  (Remote  Students  Only) 

Rev  00,  01 

STM  921.1 

None 

Tailoring  (For  Demo  Only) 

Rev  05,  0l 

STM  921.2 

STM  921 .2 

Project  Planning 

Rev  05,  oj 

STM  921.3 

None 

Software  Cost  Estimation  Overview 

Rev  02,  0'i 
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In  Process  Support 


■  Internal  assessments 

■  External  assessments 

■  Monthly  SSEPG  meetings 

■  Library  of  assets 

■  Support  to  projects  to  identify/explain  organizational 
best-practices 

■  Support  from  senior  management 

■  Set  goals 

■  Integrate  with  other  initiatives 
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Evaluations  —  Total  Life  Cycle  Performers  Need 
Help,  Easy  Access  to  Results 


Pre-Award 


Continued  process 
performance 


Focus  on  the  WHAT  ,  the  WHEN  then  the 
WHO  (FUNCTIONAL  OWNER). 

_ Pretend/operate  as  if  this  line  is  missing  ! 


oject  Execution 


Gate  Reviews  Cover 

■Requirements  Feasibility 

■Schedule  < _ 

■Technical  ^ - 


Cost/Profit 


■Past  Performance  + _ 

■Supplier  Select./Past  Perf 
■Process  +- 
■Mission  Assurance 
■Cash  Flow  <«- 
■Risks/Costed  <«- 
■Lessons  Learned 
Non-Advocate  Review 
Independent  Cost  Evaluation 


7^ 


Improved 

process 

performance 


Next  Opportunity 


•Performance  Reviews  Cover 
■Requirements  Management 
■Schedule 
^Technical 
■Cost/Profit 
■Customer  Satisfaction 
■Subcontract  Performance 
s  Process  Health 
■Mission  Assurance 
■Cash  Flow 
■Risks/Costed 
■Lessons  Learned 

•  ISO  Findings  and  Corrective  Actions 

•  CMMI  Assessments 
Metrics  from  Project  Reviews 

■Risks/Costed  Risks 
■EVM 

°  Defects  (HW/SW) 

■Subcontract  Report  Cards 
Independent  Risk  Reviews... 


t 


•Pre-award 
•Project  Execution 

•Pre-award 
•Project  Execution 


5 


Assessment  and 
evaluation  are 
continuous 
activities 
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We  Have  Optimized  Appraisals 


We  assess  in  half  the  time  of  the  industry  average 

■  Every  project  rated  against  every  CMMI  practice 

Appraiser  experience  is  key 

■  Appraisal  teams  are  6-9  people  -  over  half  the  team 
members  must  be  experienced 

■  Documented  standards  for  interpreting  sufficiency 

Preparation  and  automation  reduce  appraisal  cost  and 
time 

■  Projects  assemble  evidence  in  advance 

■  Standard  interview  questions;  templates  for  appraisal 
plan,  all  presentation 


■  Benefit:  Guidance  and  support  from  Organizational 
appraisers  save  projects  time  and  frustration 


20 


10/24/2005  1:56  PM  HEADER  /  FOOTER  INFORMATION  (SUCH  AS  PRIVATE  /  CONFIDENTIAL) 


NORTHROP  GRUMMAN 


Copyright  2005  Northrop  Grumman  Corporation 


Summary 


■  An  effective  CMMI  L5  culture: 

■  Supports  the  organization's  Vision  to  build  a 
foundation  to  deliver  consistently  high  quality 
products  and  services  that  ensure  mission  success 

■  Enhances  the  customer’s  confidence  that  projects 
have  the  Infrastructure  needed  to  meet/exceed 
contract  requirements  and  mission  needs 

■  Provides  the  Controls  needed  to  quantitatively 
manage  and  enhance  project  process  improvements 
and  excellence 


Ensures  the  discipline  necessary  in  an  environment  of 
highly-complex,  mission-critical  systems  through 
vision,  infrastructure  and  control. 
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Customer  Success  Is  Our  Mission 

Keeping  the  Team  Motivated  for 

Success 

Raytheon  Missile  Systems 
Mike  Scott  and  Mike  Notheis 


November  2005 


Introduction 


Missile  Systems 

Process  initiatives  are  hard  and  so  is  attracting  and 
keeping  talented  people  for  the  duration. 

•  This  presentation  discusses  RMS’s  approach  to: 

-  Setting  the  goal  and  vision 

-  Team  building 

-  Rewards  and  recognition 

-  Achieving  success 


How  do  you  get  over  95%  of  a  team  wanting  to  stay  on 

for  the  next  process  initiative? 
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Setting  The  Goal  &  Vision 


Missile  Systems 


Leadership  must  set  the  goal  and  the  vision  for  the  initiative 

-  The  initiative  was  about  improving  the  enterprise  and  the  way  we  do 
business 

This  was  reinforced  throughout  our  18  month  quest 

-  Leadership  established  RMS  wide  goals 

Business  goals 
Program  performance  goals 
Process  improvement  goals 


Set  the  vision  and  the  goal 
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Develop  the  Plan  &  Focus  the  Task 


Missile  Systems 


Exactly  what  tasks  need  to  be  done  -  Critical  Chain  Mgmt. 

Executive  Manager  -  Reports  directly  to  Business 
President  and  Executive  Team 

Program  Manager  -  Chief  Barrier  Remover 

Chief  Engineer  -  Lead  the  Technical  Accomplishment 

The  Team  -  Let  them  do  what  they  do  best  -  Accomplish 


Define  the  roles  and  responsibilities 
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Alignment  is  a  Key 


Missile  Systems 


Management  Team  met  at  7:30  Each  Morning  -  15  to  30 
Minutes 

The  entire  team  met  every  morning  at  8:00  for  a  15  to  30 
Minute  Stand-up 

The  Stand-up  centered  on  the  Critical  Path  only  - 
Identification  and  removal  of  barriers 


Daily  communication  to  keep  on  track 
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Building  The  Team 


Missile  Systems 


Leveraged  process  expertise  from  our  software  community 

-  These  folks  seeded  our  team 

We  needed  a  lot  more  people  power 

-  Solicited  support  from  all  engineering  and  support  organization 
disciplines 

-  Raided  our  Six  Sigma  organization 

-  Our  team  grew  to  60  full-time  people 

Highly  motivated  individuals 
People  who  were  dragged  in 
Everything  in-between 


Next  you  need  resources 
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Developing  the  CMMI  Knowledge 


Missile  Systems 


Most  of  the  team  knew  nothing 
about  CMMI 

-  Established  an  extensive  training 
plan  on  the  model 

-  Assigned  two-person  teams  to 
each  process  area 

Workshops  held 
Shoulder-to-Shoulder  reviews 

-  Process  Area  experts  cross- 
trained  rest  of  the  team 

-  Engaged  our  external  appraisal 
team  early  and  often  to  leverage 
their  expertise 


r 
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Rewards 


^ayliiBon 

Missile  Systems 


•  Rewards  must  be  meaningful 

-  We  used  typical  rewards 

Merit  and  promotion 
Team  awards 
Individual  achievement 

-  We  also  used  alternative  rewards 

Gift  certificates  to  the  local  mall  (on  the  spot) 
Maintained  a  “snack  shack”  with  drinks  and  junk 
Handed  out  badge  lanyards,  team  shirts,  etc. 

-  We  rewarded  in  other  ways 

Conference  and  seminar  attendance 
Briefing  to  executive  leadership  opportunities 
Expanded  responsibilities 


Reward  great  performance  often 
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Recognition 


Missile  Systems 


Got  to  know  our  team  as  individuals 

-  What  is  going  on  in  their  lives 

-  What  stresses  are  they  under  that  could  affect  performance 

-  What  motivates  and  de-motivates  them 

-  What  will  challenge  them  and  what  will  overwhelm  them 

Publicly  celebrated  team  and  individual  success 

-  Daily  stand-up  sessions  to  share  status  and  information 

Applauded  every  task  completion 

Celebrated  every  birthday 

Thanked  the  individual  and  team  for  each  success 

-  Luncheons  to  celebrate  milestones  achieved 


Recognize  people  in  open  forums 
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Achieving  High  Performance 


laytCason 

Missile  Systems 


Valued  high  performance  and  success  at 
team  and  individual  levels 

-  Focused  on  results 

Individual 

Team 

-  Treated  “people  as  people” 

Created  an  environment  for  success 

-  Respect  for  one  another 

-  Simple  amenities 

-  Effective  rewards  and  recognition 


PEOPLE  will  make  you  successful 
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The  Results 


Missile  Systems 


Built  a  high  performing  team  quickly  that  met  all  initiative 
milestones 

Achieved  our  ultimate  goal  on-schedule  and  under-budget 
An  easy  team  to  manage 

Over  95%  of  the  team  members  expressed  their  desire  to 
stay  together  as  a  team 


How  do  you  get  over  95%  of  a  team  wanting  to  stay  on 
for  the  next  process  initiative?  -  Treat  people  as  people! 
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Customer  Success  Is  Our  Mission 


Barrier  Busting”  -  Obtaining 
Active  Leadership  Support 

Raytheon  Missile  Systems 
Mike  Scott  &  Eric  Ziegler 

November  2005 


Introduction 


Missile  Systems 

Barriers  impede  performance.  Having  the  right 
environment  that  focuses  on  the  removal  of  these 
barriers  can  help  ensure  success. 

•  This  presentation  addresses: 

-  Establishing  and  communicating  clear  goals 

-  Having  the  right  sponsors  in  the  game 

-  Setting  up  a  leadership  structure  that  works 

-  Constant  communication  to  all  the  stakeholders 

-  Having  a  team  that  is  focused  on  success 


To  maintain  speed  and  agility  you  must  identify  and 

remove  barriers  quickly 
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Clear  Goals 


Missile  Systems 


Executive  leadership  set  measurable  goals  and  a  vision  for 
our  process  initiative 

-  The  initiative  was  about  improving  the  enterprise  and  the  way  we  do 
business 

This  was  reinforced  throughout  our  18  month  quest 

-  Enterprise  wide  goals 

Business  goals 
Program  performance  goals 
Process  improvement  goals 

Goals  socialized  and  accepted  throughout  the  organization 


Clearly  communicated  goals  get  everyone  on  the  right 

road 
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Missile  Systems 


•  Site  President 

-  Set  Vision  and  Goal 

-  Quarterly  Reviews 

-  Weekly  with  the  Executive  Interface 

•  Executive  Advisors  Group 

-  VPs  from  Engineering,  Quality,  Finance, 
Operations 

-  Twice  monthly  reviews 

•  Executive  Interface 

-  Full-time  assignment  to  the  team 

-  Chief  Barrier  Buster 


Participatory  Sponsorship  is  crucial 


Sponsorship 
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Leadership 


Missile  Systems 


Sets  the  vision  and  goals.  Breaks 
down  ENTERPRISE  BARRIERS. 

Provides  interface  to  enterprise 
executives.  Breaks  down 

ORGANIZATIONAL  BARRIERS. 

Makes  programmatic  decisions  and 
direction.  Breaks  down  PROGRAM 
BARRIERS. 

Responsible  for  all  technical 
decisions  and  direction.  Break 
down  TECHNICAL  BARRIERS. 


Leadership  sets  the  expectation  and  must  provide  the 

behavior  example  for  the  team 
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Communication 


siiaytioson 

Missile  Systems 


•  Up 

-  Quarterly  with  Site  President 

-  Twice  monthly  with  Executive  Advisors  Group 

-  Daily  with  Executive  Interface 


•  Across 

-  Twice  monthly  Program  Manager  Lunch 

-  Twice  monthly  Functional  Manager  breakfast 

-  Deployment  leads  assigned  to  each  focus 
program 


•  Down 

-  Weekly  information  sharing 

-  Daily  status  and  barrier  identification/removal 


Constant  and  clear  communication  Keeps  everyone 
vectored  in  the  same  direction 
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•  Detailed  plans  at  the  task  level  focus  the  team 

•  Daily  Stand-ups 

-  Status  completions 

-  Identify  risks 

-  Identify/resolve  barriers  (everyone  felt  comfortable  bringing  issues  to  the 
table) 

-  Immediate  corrective  action 

-  Meaningful,  daily  metrics 


A  team  that  is  focused  on  results  will  be  successful 
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The  Results 


Missile  Systems 


Identification  and  removal  of  barriers  was  issue  focused  not  punitive 

We  all  owned  and  participated  in  Barrier  Busting 

No  barrier  remained  on  the  list  for  more  than  a  week 
-  Nearly  all  resolved  in  the  same  day 

Empowered  Teams  that  learned  to  break  down  their  own  barriers 
Achieved  all  initiative  goals  on-schedule  and  under  budget 


To  maintain  speed  and  agility  you  must  identify  and 

remove  barriers  quickly 
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Space  and  Airborne  Systems 


Enterprise  Process  Integration 
within  the  Space  and  Airborne  Systems 
Business  Area  of  Raytheon 


Linda  Kovar 
and 

Deana  Seigler 
November  16,  2005 
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Setting  the  Stage 


Space  and  Airborne  Systems 


•  Raytheon  is  an  industry  leader  in  defense  and  government 
electronics,  space,  information  technology,  technical  services,  and 
business  aviation  and  special  mission  aircraft. 

-  The  company  is  divided  into  seven  major  business  units 


Space  and  Airborne  Systems  (SAS)  is  one  of  the  seven  business 
units  that  make  up  Raytheon 

-  Conglomeration  of  programs  from  various  legacy  defense  companies 
such  as  Hughes  Aircraft,  Texas  Instruments  and  Raytheon  Company 

-  2004  Revenue  of  $4.1  Billion 

-  13,000  employees 

-  4  geographic  locations 

•  El  Segundo,  CA 

•  Goleta,  CA 

•  Texas 

•  Mississippi 


Goleta,  CA  — 

El  Segundo,  CA 
- SAS  HQ 


Dallas,  McKinney, 
and  Plano,  TX 


Forest,  MS 


©  2005,  Raytheon  Company.  All  rights  reserved 


2 


Setting  the  Stage 


Space  and  Airborne  Systems 


•  Each  location  had  their  own  set  of  processes,  process 
improvement  initiatives  and  goals 

-  El  Segundo  had  been  previously  assessed  at  CMMI  Level  3  for 
Systems  and  Software  Engineering 

-  Texas  had  been  previously  assessed  at  CMMI  Level  5  for  Software 
Engineering  and  CMMI  Level  3  for  Systems  Engineering 

•  At  the  beginning  of  2004,  there  were  three  sets  of  processes  being 
developed  and  maintained  within  SAS 

-  Separate  process  groups  working  independently 

•  Site  specific 

•  Discipline  specific 
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Case  for  Action 


Space  and  Airborne  Systems 


•  Increasing  business  need  to  share 
work  between  geographic  locations 

•  Discipline-specific  processes 
existed  for  Systems,  Software,  and 
Hardware  Engineering 

-  Across  the  sites  we  found  we  had 
separate  but  similar  processes 

-  As  Hardware  Engineering  started 
down  the  process  improvement 
journey,  we  realized  many  of  the 
same  processes  would  be  needed 

•  Multiple  CMMI  appraisals  would  be 
needed  and  were  planned  due  to 
the  varying  processes  and  goals 
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The  Goal 


Space  and  Airborne  Systems 


•  In  July  2004,  Jack  Kelble,  SAS  President,  made  the  strategic 
decision  to  integrate  development  processes  across  SAS 

-  El  Segundo  already  had  a  process  architecture  called  the  Enterprise 
Management  System  (EMS) 

-  Only  the  El  Segundo  processes  were  integrated  into  this  architecture 


•  Goal  was  to  achieve  CMMI  Level  5  for  Systems  and  Software 
Engineering  and  CMMI  Level  3  for  Hardware  Engineering  in  2005 


-  As  part  of  this  goal,  all  engineering 
development  processes  were  to  be 
merged  and  integrated  into  EMS 

-  In  addition,  one  CMMI  Class  A 
appraisal  was  to  be  conducted 
for  the  entire  SAS  organization 


©  2005,  Raytheon  Company.  All  rights  reserved 


5 


Plan  of  Attack 


Space  and  Airborne  Systems 


*  Execute  this  enterprise  process  integration 
effort  like  a  program 

*  Determine  an  approach  that  would  allow 
SAS  to  integrate  processes  across  the  entire 
organization  in  a  very  short  period  of  time 

*  Develop  a  proposal  describing  how  to 
accomplish  the  goal  and  identifying  what 
resources  would  be  required 


•  Pull  the  “best  of  the  best”  processes  from 
across  SAS  to  form  the  SAS  standard 
process 

•  Create  discipline-independent  processes 
whenever  feasible 
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Look  for  better 
solutions! 


Plan  of  Attack 


Space  and  Airborne  Systems 


•  Organize  several  teams  to  develop  the  plan  to  integrate  the 
processes  across  the  enterprise 

-  Core  Proposal  Team 

-  Numerous  Mini-Teams 

-  Management  Review  Team 


Create  a  unified  Enterprise  Process 
Group  (EPG)  for  all  sites  and  disciplines 

-  Ensure  representation  from  all  sites 
on  all  teams  and  throughout  the  EPG 
Leadership  Team 

-  Reduce  process  improvement  effort  by 
maintaining  only  one  set  of  processes 
and  conducting  a  unified  appraisal 
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Core  Proposal  Team 


ReytflMHiii 

Space  and  Airborne  Systems 


•  Membership 

-  Key  process  leaders  from  each  site 

•  Responsibilities 

-  Provide  the  overall  roadmap  for  the  proposal 

-  Identify  complete  list  of  existing  processes 

-  Develop  initial  recommended  list  of  discipline-independent  processes 

-  Divide  the  process  list  into  numerous  mini-teams  by  topic 

-  Determine  common  terminology  to  be  used  for  the  SAS  Directives 

•  Procedure  versus  Directive 

•  Work  Instruction  versus  Procedure 

-  Secure  resources  to  work  mini-team  reviews 

-  Establish  process  for  mini-teams  to  review  processes 

-  Review  recommendations  and  estimates  generated  by  the  mini-teams 

-  Roll-up  estimates  and  present  plan  to  management 
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SAS  Directive  Structure 


Space  and  Airborne  Systems 


•  Policy:  Directive  and  establishes  the 
commitment  that  cannot  be  tailored. 

•  Bulletins:  Used  to  augment  policy  for  a  short 
time  or  for  frequently  changing  needs. 

•  Procedures:  Directive  and  may  not  be 
tailored.  Contain  detail  on  “What  to  do". 

•  Work  Instructions:  Directive  and  may  be 
tailored.  Contain  detail  on  “How  to  do". 

•  Enablers:  Not  directive.  Enablers  are 
provided  to  support  implementation  of 
Procedures  and  Work  Instructions. 

-  Enablers  are  samples,  templates, 
checklists,  etc.  for  what  should  be 
considered  when  performing  a  task. 


Hierarchy 


Procedures 


Work 

Instructions 


Enablers 


▼ 
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Mini-Teams 


Space  and  Airborne  Systems 


•  Membership 

-  Subject  Matter  Experts  from  each  site  for  the  various  process  areas 

-  Multi-site  representation  was  key  to  the  success  of  the  mini-teams 

•  Responsibilities 

-  Meet  (virtually)  with  the  representatives  from  each  site  to  review  the 
existing  processes 

-  Develop  a  recommendation  on  the  path  forward  for  the  specific 
process  area 

•  Keep  one  site’s  existing 
process  as  is 

•  Merge  existing  processes 
from  all  sites 

•  Eliminate  the  process 

•  Elevate  the  process  to  be 
discipline-independent 

-  Generate  detailed  Basis  of  Estimate  (BOE)  to  document  the  effort 
required  to  accomplish  the  recommendation  of  the  team 
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Space  and  Airborne  Systems 


Example  Mini-Team 


•  One  mini-team  was  assigned  the  Peer  Review  Process 

-  Subject  Matter  Experts  on  the  existing  processes  were  identified 

•  Current  State 

-  SE  Peer  Review  Directive  and  Procedure  in  Texas 

-  SW  Peer  Review  PRG  and  Procedure  in  Texas 

-  Separate  SE  and  SW  Peer  Review  Work  Instructions  in  El  Segundo 

-  Five  enablers  in  Texas  and  three  enablers  in  El  Segundo 

-  HW  did  not  yet  have  a  Peer  Review  process  at  either  site 

-  Defect  Logger  Tool  (Access  database)  used  in  Texas  and  Integrated 
Project  Reporting  Tool  (Excel  spreadsheet)  used  in  El  Segundo 

•  Recommendation 

-  Form  one  discipline-independent  Peer  Review  process 

•  Common  definition  of  a  defect  and  common  set  of  codes  for  defect 
classification  (type,  reason  and  priority) 

•  Common  program  phases  for  defect  containment 

•  Create  an  alternative,  less  formal  process  for  Desk  Checks 

-  Deploy  the  Defect  Logger  Tool  to  all  geographic  locations 
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Management  Review 


Space  and  Airborne  Systems 


•  Membership 

-  SAS  President  and  VP  of  Engineering 

•  Approve  the  budget  for  the  plan 

-  Functional  line  management 

•  Approve  the  technical  approach 

•  Responsibilities 

-  Review  and  approve  the  plan  presented  by  the  Core  Proposal  Team 

-  SAS  President  and  VP  of  Engineering  reviewed  the  budget  and  ability 
of  the  plan  to  meet  the  goal 

-  Functional  line  management  reviewed  the  recommendations  of  the 
mini-teams  to  ensure  they  were  aligned  with  the  recommendations 


©  2005,  Raytheon  Company.  All  rights  reserved 


12 


Proposed  Changes 


Space  and  Airborne  Systems 


J  (ENTERPRISE  SOLUTION 


BIN -  33.54  (+0.83) 


Space  and  Airborne  Systems  (SAS) 


Raytheon 
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Discipline-Independent 

Processes 


Space  and  Airborne  Systems 


•  A  key  goal  of  our  process  merger  effort 
was  to  replace  discipline-specific 
processes  with  discipline-independent 
ones  wherever  possible 

-  Discipline-independent  processes  are 
referred  to  as  “common”  processes 

•  Benefits  include: 

-  Reduces  the  number  of  processes  to 
maintain 

-  Facilitates  common  execution  of  process 
across  all  disciplines 

-  Allows  integrated  teams  to  talk  the  same 
language 
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Discipline-Independent 

Processes 


Space  and  Airborne  Systems 


•  Process  Tailoring 

-  Describes  how  programs  will  perform  tailoring, 
including  both  discipline-independent  and 
discipline-specific  processes 

•  Program  Planning 

-  Created  a  process,  called  the  Program 
Management  Plan,  for  the  program-level 
planning  elements 

-  Kept  discipline-specific  processes  for  details  of 
planning  requirements  by  discipline 

•  Systems  Engineering  Management  Plan 

•  Hardware  Development  Plan 

•  Software  Development  Plan 

•  Standardized  on  a  3-phase  tailoring  and 
planning  approach  for  all  disciplines 
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Discipline-Independent 

Processes 


Space  and  Airborne  Systems 


Project  Measurement  &  Analysis 

-  Used  to  help  the  program  establish  their  metrics  plan 

Team  of  X 

-  This  is  an  interactive  meeting  between  program 
personnel  and  line  management  to  review  program 
metrics,  status,  issues,  processes 

Integrated  Management  Review 

-  This  is  a  periodic  review  with  higher  level  management 
that  can  involve  more 
than  one  discipline 
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Discipline-Independent 

Processes 


Space  and  Airborne  Systems 


ENTERPRISE  SOLUTION 


Structured  Decision  Making 

-  Process  for  making  formal  decisions  that  could 
have  a  significant  impact  to  the  program 

Risk  and  Opportunity  Management 

-  Describes  how  to  identify,  categorize  and  manage 
risks  and  opportunities  for  all  disciplines 


•  Work  Product  Management  and  Stakeholder  Involvement 

-  One  matrix  that  lists  the  program’s  work  products,  level  of  control  for 
each,  stakeholder  involvement  for  each  and  designates  which  work 
products  must  be  reviewed  using  the  Peer  Review  process  x 

•  Cost  Estimation 

-  Originally  thought  to  be  disciple-specific,  but  later 
determined  it  could  be  discipline-independent 

-  Still  under  development,  but  a  new  version  to  be  piloted  soon 
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Discipline-Independent 

Processes 


Space  and  Airtrome  Systems 


Project  Teaming 

-  Describes  the  establishment  of  integrated 
product  teams 

Peer  Review  and  Desk  Check 

-  Peer  Review  process  meets  the  requirements 
of  the  CMMI  model 

-  Desk  Check  process  is  a  less  formal  process  that  can  be  used 


Gate  Reviews 

-  This  is  an  independent  review  of  the  program 
at  major  phase  transitions 

Objective  Evaluation 

-  Process  and  product  audits  by  independent  evaluators 
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OneSAS  EPG 


Space  and  Airborne  Systems 


•  The  plan  of  attack  included  unifying  the  various  process  groups 
across  the  business  into  a  single  Enterprise  Process  Group  (EPG) 

-  The  new  structure  was  referred  to  as  the 
OneSAS  EPG  to  make  it  obvious  that  we 
were  unifying  the  process  groups  and 
the  processes  into  one 

-  Created  a  logo  for  the  enterprise  process 
integration  effort 


•  The  OneSAS  EPG  would  include  representation  from  all  disciplines 
and  sites  and  would  be  responsible  for  executing  the  process 
merger  plan 

-  A  distributed  team  makes  coordination  and  communication  more 
difficult 

-  The  OneSAS  EPG  meets  weekly  via  teleconference  and  Sametime 

-  Meet  face-to-face  for  all  planning  activities  and  once  a  month  as  a 
leadership  team 
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Space  and  Airborne  Systems 


OneSAS  EPG  Organization 


•  Implemented  an  Integrated  Product  Team  (IPT)  structure  for 
process  development  and  a  Cross  Product  Team  (CPT)  structure 
for  activities  that  cut  across  all  IPTs. 


EPG 


Process  Integration  Technical  Directors: 


Robert  Gonzalez  and  John  Peyton 


Linda  Kovar, 
Program  Manager 


Enterprise 
Management 
System  CPT 

Alcantar 


Appraisal 
Coordination  CPT 

De  Cicco 


Measurement 
and  Analysis 
CPT 

Luke 


Learning  CPT 

Adams 


Process 
Improvement 
Rollout  CPT 

Raymond 


IPCCB 


PIR 


HDW  IPt 

Martin 

Heer 


SYS  IPT 

Boswortf 

Perkowsk 


SW  IPT 

Seigler 

Chacon 


PM  IPT 

Probst 


CM/DM  IPT 

Brantley 


Making  sure  processes  are  consistent  across  disciplines,  across  programs  and  with  the  architecture 
EMS/IPDS .  Address  issues  related  to  process  compliance  within  CMMI  interpretations. 


B 

=1 

0 

XT  n  H 

Making  sure  data  archiving  and  repositories  are  consistent  across  disciplines,  across  programs 
and  With  the  architecture  EMS/IPDS  .  Planning  and  collecting  artifacts  for  appraisals 


Coordinating  consistent  metrics  across  SAS.  Keeping  track  of  business  needs  and  translating  those  to  action-metrics 


Making  sure  the  training  program  is  consistent  across  SAS  organizations 


Making  sure  process  rollouts  are  consistent  across  disciplines,  across  programs.  Coordinate  with  IPT 
leads  for  deploying  the  processes.  Program  Contact  Coordination  using  the  Team  of  X.. 
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OneSAS  EPG  ConOps 


Space  and  Airborne  Systems 


Developed  concept  of  operations  (ConOps) 
for  the  IPTs  and  CPTs  to  define  the 
interactions  between  them 

-  One  generic  ConOps  for  the  discipline  IPTs 

-  Five  specific  ConOps  for  each  of  the  CPTs 


•  In  addition,  the  following  ConOps  were  needed  for  specific  tasks 

-  Top-level  EPG 

-  Process  Definition 

-  Process  Support 

-  Integrated  Process  Change  Control  Board  Change  Process 

-  Enterprise  Management  System  Website 

-  Process  Improvement  Roll-out 

-  Artifact/Data  Collection 

-  Artifact  Gap  Closure 
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OneSAS  EPG  Services 


Space  and  Airborne  Systems 


Created  a  chart  showing  the  inputs  and  outputs  from  the  EPG  to 
describe  the  services  offered  by  the  EPG 


CR 

Process  Improvement  Ideas 

Tailoring  Reports 
Lessons  Learned 


Request  for  New 
Process  Training  Course 
Request  for 
Process  Training  -  IDP 


Appraisal  Artifacts 


SAS  Business  Goals 
Metrics 


EPG 


EMS 


PIRs 


Learning 


Appraisals 


Metrics  &  Analysis 


EMS  Directives  and  Tools 

EMS  Status  of  Changes 

Communications  (EPG  Activities) 

PAL  (Lessons  Learned,  Best 
Practices,  Examples) 

Piloting 

PIR  Communication 
&  Package 

Training  Plans 
Disposition  of  Training 
Request 

Enterprise  Objective  Evaluation  Reports 
Appraisal  Program  Support 
Appraisal  Results 
Verified  Artifacts 
&  Corrective  Action  Guidance 

Org  Measurement  Plan 
Process  Capability  &  Outliers 


Request  for  Non- 
Appraisal  Program 
Support 


Services  for  Hire 
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Did  it  Work? 


Space  and  Airborne  Systems 


•  The  OneSAS  EPG  team  was  formed  and  worked  very  well  together 

-  Representation  from  each  site  and  monthly 
face-to-face  meetings  were  keys  to  our  success 

•  All  the  discipline-independent  processes  discussed  previously  are 
released  and  are  being  used  with  the  exception  of  Cost  Estimation 

-  Late  decision  to  make  Cost  Estimation  discipline-independent 


•  SAS  Achieved  CMMI  Level  3  for 
Systems,  Software  and  Hardware 
Engineering  in  August  of  2005 

-  This  multi-site,  multi-disciplined 
appraisal  was  the  largest  in  scope 
for  any  business  in  Raytheon 

-  It  was  the  first  CMMI  appraisal 
to  include  Hardware 


CMMI 


Rayttieon 

Space  and  Airborne  Systems 


'Thr  Center  for  Systems  Management  completed  a  baaed 

appraisal  an  August  30 , 2005,  in  accordance  with  the  Software 
’Engineering  Institute  VlJ;  'Method  ‘Definition 

(Document  and  it  ions  de  termined  that 

Raytheon  -  Space  and  Airborne  Systems 

exhibited  the  characteristics  of 

SEI  Level  3  Process  Maturity 
in  Systems,  Software  and  Hardware  Engineering 

as  defined  %  the  SEI  CMiMl®  ‘Version  1.1 
SE/SW  Staged  'Representation. 

CSM 


TftfS- 


Aihftri  J  TrunSxtutii  e 

SEi  Authorized  Lead  Aypr&tsvr 
Th*  C  vinter  for  Systems  Managvrnenl 


THI 

CENTER  FOR 

SYSTEMS  MANAGEMENT 
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U.  S.  Air  Force 


Integrity  -  Service  -  Excellence 


Looking  for 
Transition  in  All 
the  Wrong  Places 


U.S.  AIR  FORCE 


96th  Communications  Group 

Eglin  Air  Force  Base,  Florida 
16  Nov  2005 


Sunshine  State 


U.S.  AIR  FORCE 


Integrity  -  Service  -  Excellence 


Outline 


U.S.  AIR  FORCE 


■  Purpose 

■  Goal 

■  Scope 

■  Transition  Process 

■  Results 

■  Lessons  Learned 

■  Conclusion 


Integrity  -  Service  -  Excellence 


Purpose 


U.S.  AIR  FORCE 


■  Communicate  an  effective  method  for 
transitioning  new  groups  into  an 
established  Organization  Software  Process 
(OSP) 

■  Share  process  improvement  experiences 
and  lessons  learned  with  other 
organizations 


Integrity  -  Service  -  Excellence 


Goal 


U.S.  AIR  FORCE 


■  Expand  Process  Improvement  using  an 
Effective  Method  by  Leveraging  from 
Established  Processes 

■  Identify  Required  Process  and  Tool 
Modifications  to  Support  New  Groups 


Integrity  -  Service  -  Excellence 


Goal  (continued) 


U.S.  AIR  FORCE 


■  Apply  Lessons  Learned  from  Existing 
Software  Groups 

■  Institutionalize  Optimizing  Processes  into  a 
New  Group  within  18  Months 


Integrity  -  Service  -  Excellence 


Organizational  Scope 


U.S.  AIR  FORCE 
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Integrity  -  Service  -  Excellence 


Scope 


U.S.  AIR  FORCE 


■  Organization  Achieved  CMM®  Level  5  with 
6  Target  Software  Groups  Defined 

■  Primarily  Software  Development  and/or  Maintenance 

■  Transition  New  Software  Group 

■  50%  Software  Development  and/or  Maintenance 

■  50%  Systems  Administration  Support 


Integrity  -  Service  -  Excellence 


Scope  (continued) 


U.S.  AIR  FORCE 


■  Transition  7  Systems  Groups 

■  Migrate  Software  and  Systems  Groups  to 
Software  and  Systems  Capability  Maturity 
Model  Integration®  (CMMI®) 

■  Transition  Services  Group  based  on  Services 
CMMI® 


Integrity  -  Service  -  Excellence 


Transition  Process 


U.S.  AIR  FORCE 


■  Execute  Orientation 

■  Establish  Training  Plan 

■  Identify  Transition  Activities 

■  Implement  Transition 

■  Collect  Measurements 


Integrity  -  Service  -  Excellence 


Transition  Process  - 
Execute  Orientation 


U.S.  AIR  FORCE 


■  Identify  Support  Infrastructure 

■  Identify  Transition  Team  Members 

■  Update  Documentation 

■  Charters 

■  Policies 

■  Communicate  Transition  Partner  Activities 


Integrity  -  Service  -  Excellence 


Transition  Process  - 
Execute  Orientation  (cont) 


U.S.  AIR  FORCE 


■  Create  a  Transition  Package 

■  Conduct  Orientation  Briefing 

■  Establish  Meetings 

■  Monthly  Transition  Status  Meetings 

■  Weekly  Transition  Working  Meetings 


Integrity  -  Service  -  Excellence 


Transition  Process  - 
Establish  Training  Plan 


W 


U.S.  AIR  FORCE 


■  Coordinate  Transition  Partner  Support  Activities 

■  Quarterly  User  Group  Meetings 

■  Monthly  Senior  Management  Review  Meetings 

■  Weekly  Technical  Working  Group  Meetings 

■  Weekly  Software  Engineering  Process  Group  (SEPG) 
Meetings 

■  Execute  Training  Process  to  Create  an  Individual 
Training  Matrix  Form  (ITMF)  for  Each  Team 
Member 


Integrity  -  Service  -  Excellence 


■ 


J0.  Transition  Process  - 

V  5P  Establish  Training  Plan  (cont) 


U.S.  AIR  FORCE 


■  Create  a  Training  Plan  Based  Upon  Team  Member 
Expertise 

■  Execute  Training  Plan  Based  on  Defined  Approach 

■  Block  Learning  Approach  -  Requires  the  Student  be  Proactive  in 
Learning  and  Applying  the  Training  Skills 


■  Spiral  Learning  Approach  -  Requires  the  Instructor  be  Proactive 
in  Teaching  Concepts  that  Build  Upon  Each  Process  and 
Applying  Real-World  Examples  to  Develop  the  Trainir  for 

Each  Student 


Integrity  -  Service  -  Excellence 


W  M  Transition  Process  - 

PP"  Establish  Training  Plan  (cont) 

U.S.  AIR  FORCE 


■  Update  Individual  Training  Matrix  Forms  (ITMFs) 
Based  Upon  Completed  Training  Courses 


■  Distribute  Updated  ITMFs  to  the  Organization 
Training  Manager  to  Update  the  Training 
Database 


Integrity  -  Service  -  Excellence 


U.S.  AIR  FORCE 


Transition  Process  - 
Identify  Transition  Activities 


■  Document  the  Meeting  Minutes 

■  Distribute  the  meeting  minutes  to  Relevant 
Stakeholders  to  Communicate  the  Status 


Integrity  -  Service  -  Excellence 


U.S.  AIR  FORCE 


Transition  Process  - 
Identify  Transition  Activities  (cont) 


■  Review  Each  Process  Step  in  the 
Organizational  Software  Process  (OSP) 


Identify  which  Process  Steps  are  Executed  in  the 
Transition  Group 


Create  Initial  Metrics  to  Document  the  Number  of 
Process  Steps  Currently  Executed  in  the  Group 


Integrity  -  Service  -  Excellence 


Transition  Process  - 
Identify  Transition  Activities  (cont) 


U.S.  AIR  FORCE 


■  Assign  Complexity  to  Each  Step  using 
(Role  Factor  x  Process  Factor) 

■  Role  Factor  -  Who  Performs  the  Step 

■  1  =  Executive  Steering  Committee,  Senior  Management,  SEPG,  Organization 
Training  Manager,  Organization  Software  Quality  Assurance  (QA)  Manager,  or 
Project  Support  Office 

■  2  =  Project  Quality  Assurance  Manager,  Configuration  Management  Manager, 
First  Level  Supervisor,  Group  Leader,  Configuration  Control  Board  (CCB), 
Transition  Partner 

■  3  =  CCB  Member,  Project  Leader,  Development  Team  Member 


Integrity  -  Service  -  Excellence 


Transition  Process  - 
Identify  Transition  Activities  (cont) 


U.S.  AIR  FORCE 


■  Assign  Complexity  to  Each  Step 
(Role  Factor  x  Process  Factor) 


■  Process  Factor  -  Action  Verb  in  the  Step 

■  1  =  Acquire,  Attend,  Submit,  Provide,  Update,  Add,  Coordinate,  Distribute, 
Place,  Schedule,  Notify,  Initiate,  Store,  Approve,  Reach  Consensus,  Assign, 
Send,  Proceed,  Collect,  Annotate 

■  2  =  Record,  Identify,  Document,  Consolidate 

■  3  =  Analyze,  Verify,  Execute,  Convene,  Determine,  Define,  Develop,  Conduct, 
Discuss,  Process,  Perform,  Complete,  Examine 


Integrity  -  Service  -  Excellence 


Transition  Process  - 
Identify  Transition  Activities  (cont) 


U.S.  AIR  FORCE 


■  Pre-defined  Process  Criticality  in  Transition 
Metrics 

■  C  =  Consistency  Failure  if  Step  is  not  Executed 

■  D  =  Data  Collection  Failure  if  Step  is  not  Executed 

■  P  =  Performance  Failure  if  Step  is  not  Executed 


Integrity  -  Service  -  Excellence 


U.S.  AIR  FORCE 


Transition  Process  - 
Identify  Transition  Activities  (cont) 


■  Document  and  Prioritize  Steps  not  Performed 
in  the  Group 

■  Document  Effort,  Duration,  Software,  Affected 
Personnel  and  Hardware  Resources  Based 
Upon  Complexity  Factor 


Integrity  -  Service  -  Excellence 


Transition  Process  - 
Identify  Transition  Activities  (cont) 


U.S.  AIR  FORCE 


■  Conduct  Transition  Overview  Briefing  with  the 
Team  to 

■  Review  Organization’s  Process  Improvement  Journey 

■  Train  the  Transition  Process  * 

■  Show  the  Way  Ahead 


Integrity  -  Service  -  Excellence 


Transition  Process  - 
Implement  Transition 


U.S.  AIR  FORCE 


■  Select  a  Project  within  the  Transition  Partner  Group  to 
Pilot  the  Organizational  Software  Process 

■  Identify  a  Project  Mentor  from  an  Existing  Group 

■  Acquire  Feedback  for  Process  Updates  Based  Upon 
the  Pilot  Project 
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Transition  Process  - 
Implement  Transition  (cont) 


U.S.  AIR  FORCE 


■  Submit  Change  Request  for  Feedback  Requiring 
Organizational  Software  Process  (OSP)  Modifications 
to  the  SEPG 

■  Update  the  OSP  to  Support  the  Transition  Partner 

■  Coordinate  Training  for  the  OSP  Modifications 
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Transition  Process  - 
Implement  Transition  (cont) 


U.S.  AIR  FORCE 


■  Provide  Training  Matrix  Updates  to  the 
Organizational  Training  Manager  to  Update  the 
Training  Database 

■  Pilot  the  updated  Organization  Software  Process 
(OSP)  within  an  existing  (experienced)  group 


Integrity  -  Service  -  Excellence 


Transition  Process  - 
Implement  Transition  (cont) 


U.S.  AIR  FORCE 


■  Acquire  Feedback  and  Lessons  Learned  Based 
Upon  the  Pilot 

■  Submit  the  Organizational  Software  Process 
(OSP)  Change  Request  to  the  SEPG  as 
Applicable 


Integrity  -  Service  -  Excellence 


Transition  Process  - 
Collect  Measurements 


U.S.  AIR  FORCE 


■  Document  the  Actual  Effort  (in  hours)  Expended 
to  Complete  the  Transition  Tasks 


■  Document  the  Estimated,  Planned  and  Actual 
Changes  to  the  OSP 


Integrity  -  Service  -  Excellence 


Results 


U.S.  AIR  FORCE 


■  Allocated  10%  -  20%  Effort  to  SEPG  Mentor 
assigned  for  Process  Facilitation 

■  Showed  Process  Changes  Expected  during  Initial 
Analysis;  however,  Showed  No  Process  Changes 
Required  from  Piloting  Feedback 

■  Established  a  New  Target  Software  Group  in  less 
than  1  year,  which  indicates  Transition  Process  is 
Extremely  Successful 
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Results  (continued) 
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Percentage  of  Expected  Changes  (Base  %) 
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Results  (continued) 
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As  of: 


Process  Name 


Technical 

Interchange  Meeting 
(TIM) 


Transition  Process  Complexity  Analysis  Form  (TPCAF) 


30  December  2004 

Step# 

Yes/No 

1 
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2 

No 

3 

Yes 

4 

Yes 

5 

No 

6 

No 

7 
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9 
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0 
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Criticality  Factor 

■  C  -  Consistency 

■  D  -  Data  Collection 

■  P  -  Performance 

Role  Factor  (who) 

■  1  -  Management 

■  2  -  Support 

■  3  -  Team 

Process  Factor  (verbs) 

■  1  -  Simple 

■  2  -  Average 

■  3  -  Difficult 

Step  Complexity 

■  Role  *  Process  Factors 

Process  Complexity 

■  Add  up  Step 
Complexities 

Work  Remaining 

■  Base  %  *  Process 
Complexity  Factor 


Integrity  -  Service  -  Excellence 


Lessons  Learned 


U.S.  AIR  FORCE 


■  Tailor  the  Training  Plan  Based  Upon  Personnel 
Skills  and  Experience 

■  Demonstrate  Tool  Functionality  using  an  Example 
Project  During  Training  Sessions 


Integrity  -  Service  -  Excellence 


Lessons  Learned 
(continued) 


U.S.  AIR  FORCE 


■  Communicate  Activity  Status  and 
Schedule  Changes  on  a  Periodic  Basis 

■  Identify  a  Process  Champion  to  Regularly  Mentor 
to  Other  Team  Members 


Integrity  -  Service  -  Excellence 


Conclusion 


U.S.  AIR  FORCE 


■  Expected  Transition  to  Require  Updates  to  the 
Organizational  Software  Process  and  Associated 
Artifacts,  but  we  were  “Looking  for  Transition  in 
all  the  Wrong  Places”  because 

■  Basic  Engineering  Principles  implemented  within  the 
Organization  provided  a  means  to  Utilize  Existing  Processes 

■  New  Processes  do  not  need  to  be  created  for  each  Discipline 

■  Cultural  Change  versus  Documentation/Process  Change  is  a 
More  Effective  Means  for  Transitioning 


Integrity  -  Service  -  Excellence 


Conclusion  (continued) 


U.S.  AIR  FORCE 


■  Plan  to  Utilize  the  Transition  Process  to 


■  Expand  Process  Improvement  to  Groups  with  Systems 
Engineering  (SE),  Supplier  Sourcing  (SS)  and  Integrated 
Process  and  Product  Development  (IPPD)  Disciplines 

■  Leverage  Existing  Processes  to  Reduce  the  Cycle  Time  for 
Developing  Processes  that  Support  the  Software  Engineering, 
Supplier  Sourcing,  and  Integrated  Process  and  Product 
Development 


Integrity  -  Service  -  Excellence 


Questions? 


U.S.  AIR  FORCE 
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Accelerating  Process  Improvement  through  Collaboration: 

The  NAVAIR  Systems  Process  Improvement  Community  of  Practice 


NAV^A  I  R 


Definition:  What  is  a  CoP? 


“Communities  of  Practice  are  places  of  learning  where  we  find 
out  what  others  already  know  and  can  do,  and  where  others  find 
out  what  we  already  know  and  can  do. 

□  A  CoP  has  a  subject  which  is  of  interest  to  people  with  different 
backgrounds  and  different  perspectives. 

□  A  CoP’s  prime  importance  is  attached  to  practical  experience  and 
questions.  The  members  exchange  experiences  and  jointly  search 
for  answers  to  their  questions  and  solutions  to  their  problems. 

□  A  CoP  organizes  itself:  The  members  agree  on  the  scope  and 
demarcation  of  the  subject,  work,  and  forms  of  exchange  and  joint 
products. 

□  A  CoP  must  be  fun  so  that  it  is  personally  enriching.”  (2) 


The  boundaries  of  the  CoP  are  defined  by  actual 
participation,  not  by  affiliation  or  title.  (6) 
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Accelerating  Process  Improvement  through  Collaboration: 

The  NAVAIR  Systems  Process  Improvement  Community  of  Practice 


Definition:  What  is  a  CoP? 


There  are  essentially  two  types  of  CoPs  (4): 

Self  Organizing  Self-organizing  CoPs  are  also  self  governing.  They 

pursue  the  interests  of  the  groups  members.  Due  to  their 
voluntary  nature,  they  are  fragile  yet  resilient.  They  are 
fragile  in  that  attempts  to  manage  them  can  cause  the 
members  to  disband  or  go  “underground.”  Yet  they  are 
resilient  in  that  members  come  and  go  as  interests  shift 
and  subjects  evolve. 

Sponsored  Sponsored  CoPs  are  initiated,  chartered,  and  supported 

by  management,  and  are  expected  to  produce 
measurable  results  that  benefit  the  organization. 


The  NA  VAIR  SPI  CoP  is  a  hybrid \  simultaneously  both  self¬ 
organizing  and  sponsored.  Sponsorship  from  the  NSSC  was 
critical  to  launch  and  initial  momentum,  but  there  is  no  hierarchy. 
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Rationale  and  Business  Case  for  CoPs 


There  are  compelling  reasons  and  business  cases  for 
establishing  communities  of  practice.  Some  rationale  found  by 
others  for  this  undertaking  are: 

1 .  Information  superiority  provides  the  joint  force  a  competitive 
advantage  only  when  it  is  effectively  translated  into  superior 
knowledge  and  decisions.  (1) 

2.  Uncertainty  from  downsizing,  retirements,  etc.  (1) 

3.  Reduce  time  in  meetings,  on  email,  on  phone  (1) 

4.  Need  for  greater  organizational  focus  (1 ) 

5.  Reduce  redundant  efforts  (1 ) 

6.  Need  for  faster,  better  informed  decisions 
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Accelerating  Process  Improvement  through  Collaboration: 

The  NAVAIR  Systems  Process  Improvement  Community  of  Practice 


Rationale  and  Business  Case  for  CoPs 


Compelling  reasons  and  business  cases  for  establishing 
communities  of  practice  ...  continued: 

7.  Sustainable,  competitive  advantage  (4) 

8.  Continuous  innovation  (4) 

9.  Reduce  reinvention 

10.  Avoid  repeating  same  or  similar  mistakes  made  by  others 
1 1  .Accelerated  process  improvement  and  implementation 
12.  Institutionalize  organizational  and  individual  learning 
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Accelerating  Process  Improvement  through  Collaboration: 

The  NAVAIR  Systems  Process  Improvement  Community  of  Practice 


Establishing  a  CoP  Culture 


Enablers  and  Inhibitors  to  Successful  CoPs  (1) 


Enablers 

Inhibitors 

High  trust 

Fear  and  suspicion 

Sharing  is  rewarded 

Hoarding  is  rewarded 

Team-based  work 

Individual  effort 

Process  focus 

Function  focused 

Open  to  outside  ideas 

Not  invented  here 

Time  to  share 

Too  busy  to  share 

Compatible  IT 

Incompatible  IT 

Need-to-share 

Compartmentalization 

Local  decision-making 

Central,  top-down  decisions 

N  AV^A  I  R 
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Accelerating  Process  Improvement  through  Collaboration: 

The  NAVAIR  Systems  Process  Improvement  Community  of  Practice 


NAVAIR  and  SPI  CoP  Drivers 


The  NAVAIR  software  and  systems 
community  continuously  evolves,  and 
must  find  innovative  solutions  to  the 
challenges  of  delivering  quality  systems 
to  the  Warfighter  as  effectively  and 
efficiently  as  possible. 

The  times  and  environment  demanded  change: 

□  Multiple  product  teams  in  dozens  of  locations 

□  Supporting  about  85  platforms  and  programs 

□  Seemingly  disparate  improvement  initiatives  from  executive 
leadership 

□  Increasing  demand  to  reduce  the  cost  of  software  and  systems  while 
increasing  quality  and  reducing  cycle  time 


NAV^A  I  R 
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Accelerating  Process  Improvement  through  Collaboration: 

The  NAVAIR  Systems  Process  Improvement  Community  of  Practice 


NAVAIR  and  SPI  CoP  Drivers  (cont) 

In  other  words,  the  NAVAIR  software  and  systems  organizations 
was  in  the  same  situation  as  many  other  organizations  (maybe 
even  yours). 

One  possible  solution  was  to  form  a  real  Community.  The 
formation  of  a  NAVAIR  Systems  Process  Improvement 
Community  of  Practice  (SPI  CoP)  promised  some  ways  to  meet 
the  challenges: 

□  Sharing  process  improvement  best 
practices  and  process  assets  to  avoid 
reinvention 

□  Continuously  sharing  lessons  learned  so 
they  don’t  have  to  be  re-learned 

□  Sharing  resources  to  accomplish  common 
goals  and  resolve  common  problems 


N  AV^A  I  R 
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Accelerating  Process  Improvement  through  Collaboration: 

The  NAVAIR  Systems  Process  Improvement  Community  of  Practice 


The  NAVAIR  SPI  CoP  Vision 


The  SPI  CoP  is  a  forum  for  NA  VAIR 
Software/Systems  Process 
Improvement  (SPI)  teams  to  share 
processes,  procedures,  tools,  and 
lessons  learned  to  improve  the 
timely  production  and  delivery  of 
quality  software  to  the  Warfighter. 
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Accelerating  Process  Improvement  through  Collaboration: 

The  NAVAIR  Systems  Process  Improvement  Community  of  Practice 


NAVAIR  SPI  CoP  Goals 


To  be  considered  successful,  the  NAVAIR  SPI  CoP  should  achieve 
these  goals  and  deliver  these  benefits  to  the  NAVAIR  software  and 
systems  engineering  community: 

1 .  Establish  a  NAVAIR  community  of  Process  Improvement  (PI) 
advocates  from  across  teams  and  competencies  responsible  for 
providing  software-intensive  systems  to  the  warfighter. 

2.  Provide  a  forum  for  sharing  software  /  systems  engineering  and 
management  products  such  as  processes,  procedures,  lessons 
learned,  tools,  and  code. 

3.  Create  /  use  a  repository  of  process  improvement  products. 

4.  Communicate  insights  and  products  shared  by  this  forum  back  to 
the  teams  and  competencies,  and  where  appropriate,  advocate  their 
effective  use  on  programs. 

5.  Provide  a  collective  voice  for  capturing  and  evaluating  systemic 
software/systems  process  improvement  issues  and  needs. 


NAV^A  I  R 
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Accelerating  Process  Improvement  through  Collaboration: 

The  NAVAIR  Systems  Process  Improvement  Community  of  Practice 


Brief  History  of  the  NAVAIR  SPI  CoP 


Oct  04 


Nov  04 


Jan  05 
Feb  05 


May  05 


Sep  05 


Research  conducted  to  understand  communities  of  practice; 
began  planning  charter  meeting 


First  meeting  in  San  Diego:  initial  community  representation 
established;  no  professional  facilitation;  prototype  agenda; 


e-Room  in  early  stages  of  design  and  prototyping 


Second  meeting  in  New  Orleans:  Standard  meeting  items  piloted; 
e-Room  structure  unveiled;  first  working  groups  formed; 
externally  facilitated;  meeting  evaluations  introduced 


Third  meeting  in  Annapolis;  Agenda  working  group  formed; 
training  day  added  as  standard  feature  to  meeting  agenda; 
externally  facilitated;  WGs  report  progress/status 


Fourth  meeting  in  Las  Vegas:  highest  attendance  to  date;  self- 
facilitated;  Yellow-Belt  training;  Mission  Area  Teams  leverage 
meeting 
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SPI  CoP  Successes 


The  NAVAIR  SPI  CoP  has  resulted  in  these  successes  and  benefits: 

□  Gives  boost  to  the  formation  of  a  consolidated 
SSEPGs  (Strike  MAT  and  Special  Mission  MAT) 

□  Has  saved  member  communities  significant  effort 
by  preventing  reinvention  and  rework  of  process 
assets 

□  Is  engendering  a  common  language  for  process 
improvement  among  member  organizations 

□  Puts  members  in  touch  with  expertise  and  experience  they  were  previously 
not  aware  of 

□  Enables  community  members  to  address  systemic,  community-wide 
problems  ...  no  one  has  to  “go  it  alone” 

□  Has  dramatically  increased  the  availability  of  process  assets  for  all  members 


NAV^A  I  R 
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Accelerating  Process  Improvement  through  Collaboration: 

The  NAVAIR  Systems  Process  Improvement  Community  of  Practice 


Features  of  a  NAVAIR  SPI  CoP  Meeting 


Training  Day 

SPI  CoP  meeting  double  functions  as  training  by  adding  process- 
related  training  day  to  two-day  SPI  CoP  meeting.  Reduces  NAVAIR- 
wide  training  costs  through  consolidation. 

Affirmations 

Community  members  provide  testimonials  of  benefits  to  home 
organizations’  PI  efforts  resulting  from  SPI  CoP  participation. 

Environmental  Scan 

NAVAIR  management  has  opportunity  to  address  and  discuss 
changes  and  initiatives  that  may  affect  all  Community  members. 

Souvenir  Sessions 

Community  members  present  or  demonstrate  processes  or  process 
assets  that  have  worked  in  their  home  organizations. 

Working  Group  Sessions  and  Out-briefs 

Working  Groups  have  opportunity  for  face-to-face  processing  of  goals 
and  action  items.  Report  progress  and  status  to  Community  in  same 
meeting. 


N 
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There  are  essentially  3  major  components  of  the  NAVAIR  SPI  CoP: 


Do  this,  dothat.Stop.  Do 
this,  do  that.  Do  this,  do 
that.  Do  this,  do  that.  Do 
this,  do  that.  Do  this,  do 
that.  Do  this,  do  that.  Do 
this,  do  that.  And  then  do 
this,  and  then  do  that. 

Do  this,  do  that.  Do  this,  do 
that.  Do  this,  do  that.  Do 
this,  do  that.And  then  do 
this.  Stop.  Do  this,  do  that. 


Do  this,  do  that.  Do  this,  do 
that.  Do  this,  do  that. 


People 


Technology 


Processes 


The  participation  of 
people  with  subject 
matter  expertise  who  are 
willing  to  share  their 
knowledge  with  others. 


Shared  information 
technology  that  provides 
media  for  the  exchange 
of  ideas  and  knowledge 
assets. 


Methods  and  processes 
that  enable  the 
exchange  of  ideas  and 
knowledge,  and  maintain 
the  integrity  of 
knowledge  assets. 
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Major  Attributes  of  SPI  CoP  Success 

(continued) 


People  are  the  most  important 
component  of  the  NAVAIR  SPI  CoP.  It  took 
people  to  form  the  SPI  CoP,  and  it  is  for 
people  that  the  Community  thrives. 


Some  of  the  attributes  and  qualities  the  people  in  the  NA  VAIR 
SPI  CoP: 

□  Have  knowledge,  experience,  and  expertise  in  software/systems  process 
improvement 

□  Members  are  representative  of  NAVAIR  Product  Teams,  and  are  empowered 
to  make  process  improvement  commitments  and  decisions  for  home 
organizations 

□  Have  a  positive  attitude  toward  sharing  their  expertise  with  others 

□  Make  the  time  to  participate  in  the  SPI  CoP 
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Major  Attributes  of  SPI  CoP  Success 

(continued) 


Technology  has  been  critical  to  the 
success  of  the  NAVAIR  SPI  CoP.  The 
appropriate  use  of  technology  enhances 
the  experience  of  the  participants. 


Communication  and  work  product  sharing  has  been  significantly 
enhanced  through  the  use  of  an  e-Room  (Documentum).  The  major 
enablers  are: 

□  Discussion  threads 

□  Organized  process  asset  sharing 

□  Easy,  user-friendly  document  search  based  on  standard  attributes 

□  Simple  access  control 

□  Automated  change  and  configuration  management 

□  Automated  e-Room  usage  and  performance  measures  (community  activity) 

□  Intuitive,  user-defined  database  facility 
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Major  Attributes  of  SPI  CoP  Success 

(continued) 


Processes  have  been  important  but  not 
critical  to  a  successful  SPI  CoP.  Well 
designed  and  executed  processes  ensures 
the  SPI  CoP  thrives  even  when  members 
come  and  go  from  the  community. 


The  critical  few  processes  that  have  contributed  to  the  NAVAIR  SPI 
CoP  success  are: 

□  Decision  making  processes  conducive  to  progress  and  inclusion  (DAR) 

□  Agenda  development  and  meeting  minutes/action  reporting  processes 

□  Meeting  logistics  process 

□  Processes  governing  e-Room  use 

□  Defined  measures  to  collect  and  analyze  to  determine  SPI  CoP  goal 
achievement 
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Institutionalizing  the  NAVAIR  SPI  CoP 
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Building  the  SPI  CoP  is  one  thing,  making  sure  it  thrives,  endures,  and 
continues  to  deliver  value  to  the  community  is  another.  We  are  still  working  to 
ensure  the  SPI  CoP  addresses  these  important  institutionalization  factors: 


□  Collecting  and  reporting  measures  and  other 
information  that  indicate  benefits  to  member 
organizations 

□  Rewarding  SPI  CoP  participation  and  contributions 

□  Developing  and  applying  consensus  definitions  for 
concepts  such  as  “good  ideas”  and  “best  practices” 

□  Monitoring  community  participation,  and  using 
experiential  information  and  feedback  from 
participants  to  continually  enhance  the  experience 

□  Ensuring  the  media/technology  used  to  support  the 
SPI  CoP  is  cost-effective,  easy  to  use,  and  is  driven 
by  the  needs  of  people  (not  the  other  way  around) 


Were  still  on 
the  journey. 


Accelerating  Process  Improvement  through  Collaboration: 

The  NAVAIR  Systems  Process  Improvement  Community  of  Practice 


NAV^A  I  R 


What  We  Have  Learned 


In  just  one  short  year,  we  have  learned  much  about  ourselves  and  our 
Community.  Our  advice  to  others  thinking  about  forming  a  CoP 
includes: 

1 .  Don’t  re-learn  our  lessons. 

2.  Get  members  of  the  community  involved  as  soon  as  possible  in 
developing  the  meeting  agendas;  reward  contributions. 

3.  Establish  goals,  and  continuously  review  them  and  measure 
performance. 

4.  Use  outside  experts  who  have  CoP  experience  to  give  launch 
power  to  start-up  structuring  and  planning.  But  have  weaning 
plan. 

5.  Use  professional  facilitation  for  meetings,  but  only  up  until  the 
Community  learns  to  self-facilitate.  Make  sure  facilitators  know 
content  of  Community’s  interests  and  work. 
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Accelerating  Process  Improvement  through  Collaboration: 

The  NAVAIR  Systems  Process  Improvement  Community  of  Practice 


NAV^A  I  R 


What  We  Have  Learned  (continued) 


If  you’re  planning  to  form  a  CoP  (continued): 

6.  Provide  incentives  for  participation  until  participation  becomes 
the  incentive  ...  “prime  the  pump.” 

7.  Do  not  mandate  participation. 

8.  Use  technology  to  enhance  communication,  learning,  and 
knowledge  capture. 

9.  Find  ways  to  make  progress  outside  of  meetings. 

1 0.  Make  sure  every  event  involves  tangible  take-aways. 

11.  Be  very  flexible.  Follow  the  interests  of  the  Community,  even  if 
it  leads  the  Community  to  morph  into  something  different. 

12.  Do  not  form  a  hierarchy;  try  to  keep  the  community  a  network. 

13.  Don’t  put  too  much  content  into  any  one  meeting;  people  need 
more  time  to  focus  and  address  each  topic. 


NAV^A  I  R 
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Accelerating  Process  Improvement  through  Collaboration: 

The  NAVAIR  Systems  Process  Improvement  Community  of  Practice 


What  We  Have  Learned  (continued) 


If  you’re  planning  to  form  a  CoP  (continued): 

14.  Provide  multiple  vehicles  and  avenues  for  feedback  from  the 
community  members,  and  make  it  obvious  that  feedback  is 
used. 

15.  SPI  CoP  can  be  a  very  effective  forum  for  integrating  multiple, 
unintegrated  improvement  initiatives. 

16.  Provide  initial  seed  funding  for  meetings  and  travel,  particularly 
when  community  members  come  from  many  different 
organizations. 


NAV^A  I  R 
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Accelerating  Process  Improvement  through  Collaboration: 

The  NAVAIR  Systems  Process  Improvement  Community  of  Practice 
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Accelerating  Process  Improvement  through  Collaboration: 

The  NAVAIR  Systems  Process  Improvement  Community  of  Practice 


For  More  Information  ... 
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Katie  Smith 

NAVAIR  Systems  and  Software  Support  Center  (NSSC) 

katie.smith@navy.mil 
760-939-1 696 

Michael  West 

Natural  SPI,  Inc. 
michael@naturalspi.com 
Toll  free  in  the  US:  866.648.5508 
Fax:  310.878.0501 


Accelerating  Process  Improvement  through  Collaboration: 

The  NAVAIR  Systems  Process  Improvement  Community  of  Practice 
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Accelerating  Process  Improvement  through  Collaboration: 

The  NAVAIR  Systems  Process  Improvement  Community  of  Practice 


CMMI  GP  2.8  Interpretation  and  Implementation: 
Is  The  Practice  Just  About  Numbers? 


5th  Annual  CMMI  Technology  Conference  &  User  Group  14-17  November  2005 


Agenda 


Review 
Three  Keys 

Evolutionary  Understanding 
Balancing  Variables 
Measures  For  Success 
Implementation  Pitfalls 
Appraisal  Considerations 


Measure  Process 
Performance 
Against  Goals 


Analysis  & 
Corrective  Action 


Report,  Store  and 
Feedback 
Performance  Info 
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GP  2.8  Monitor  and  Control  the  Process 


Monitor  and  control  the  process  against  the  plan  for  performing  the 
process  and  take  appropriate  corrective  action. 

S  Perform  the  direct  day-to-day  monitoring  and  controlling  of  the 
process 

s  Visibility  into  the  process  is  maintained  so  that  appropriate 
corrective  action  can  be  taken  when  necessary. 

S  Measure  appropriate  attributes  of  the  process  or  work  products 
produced  by  the  process. 


i  Refer  to  the  Project  Monitoring  and  Control  process  area  for  more 
information  about  (the  topics  of)  monitoring  and  controlling  the 
project  and  taking  corrective  action 

■  Refer  to  the  Measurement  and  Analysis  process  area  for  more 
information  about  (using)  measurement  (as  the  reporting 
mechanisms  in  preparation  for  higher  maturity  level) 
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Involves  Three  Keys 

Monitoring  Measurement  &  Analysis  Control 


Defined  as: 

The  collection, 
recording, 
tracking  and 
reporting  of 
important  activity 
information 


Example  Activities: 

P  Progress  &  status 
reporting  of  activities 
and  products 

■  Updates  to  lists  of  action 
items,  risks,  problems, 
and  issues 

■  Comparisons  of  actual 
process  data  against 
established  goals,  the 
cost  /  benefit  analysis 
used  when  establishing 
a  process 


Defined  as: 

The  development  and 
sustainment  of  a 
quantitative 
capability  to  support 
sub-process  or 
process  (later  for 
project  and 
organizational  needs) 

Example  Activities: 

i  Specifying  goals  and 
measures  to  collect 

■  Analysis  mechanisms, 
baselines  and  decision 
thresholds 

■  Comparisons  against  goals 
and  objectives 

■  Data  storage  and  retrieval 
mechanisms  (data 
management) 


Defined  as: 

Managing  changes 
and  corrective 
actions  necessary  to 
bring  actual 
performance  into 
agreement  with  plan 


Example  Activities: 

Updates  to  the  plan  and 
schedule,  to  reflect  actual 
progress 

■  Resolution  of  items  that 
were  unknown  or  that  have 
changed  since  the 
implementation  / 
modification  of  a  process 


To  watch,  keep  track  of,  or 
check  for  a  special  purpose 


Using  numbers  to  determine 
goal  satisfaction  (limits) 


To  exercise  restraint  or  direct 
influence  over:  i.e., 
replanning 
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In  An  Evolutionary  Manner,  It  Helps: 


■  Develop  the  rudimentary  mechanisms  to: 

s  Identify  what  to  collect  to  meet  needs 

S  Developing  the  capability  to  collect  the  right  data  and  document 
and  share  best  practices  for  a  process  area  or  sub-process 

Begin  to  establish  the  patterns  for  modeling  and  analysis  for  a 
collection  of  similar  capabilities  for  a  project 

■  Set  the  stage  for  understanding  of  the  current  strengths  and 
weaknesses  of  the  organization's  processes  and  process  assets 

■  Continue  to  support  data  needs  for  the  advanced  capability  to 
achieve  quantitative  project  and  organizational  objectives  for  quality 
and  process  performance  through 

S  Common  measures 
s  Process  performance  baselines 
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Additionally,  It  Is  About 


Balancing  these  variables: 

S  Needs  -  urgent  want  or  necessity  arising  from  circumstances 

S  Verification  -  is  it  (or  isn't  it)  satisfying  process  requirements 
based  on  needs 

s  Change  -  do  we  or  don’t  we  change  the  process  based  on 
outcomes  and  variations  compared  against  needs 


“Process  will  always  affect  Project  Performance!” 
P.  Lewis,  Project  Planning,  Scheduling  and  Control 
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What  Balance  Translates  Into 


■  Improving  process  performance  together  with  management  of  the 
project 

Revealing  problems  early  so  that  action  can  be  taken 

Ensuring  the  quality  of  project  work  (e.g.  process  useage)  does  not 
take  a  back  seat  to  schedule  and  cost  concerns 

Verifying  that  in-place  processes  are  used  correctly  (via  expected 
outputs) 

■  Identifying  areas  where  other  project  /  process  segments  should  be 
managed  differently  (what  we  did  doesn't  fit) 

■  Keeping  clients  /  stakeholders  informed  of  process  /  product  / 
project  status 


■  Reaffirming  the  organizations  commitment  to  the  project 

(continuous  alignment  with  goals  and  objectives)  [for  the  benefit  of 
the  team  and  stakeholders] 


Ensuing  that  the  established  process  is  retained  during  times  of  stress 
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Process  Data  Collecting 


i  Data  collected  and  reported  should  fall  into  these  categories: 

s  Frequency  counts  per  time  period  -  e.g.  defects  per 
thousand 

■s  Raw  numbers  in  ratio  -  actual  amounts  used  /  produced 
against  a  limit 

■s  Subjective  numeric  ratings  -  ordinal  rating  of 
performance  but  can’t  be  mathematically  processed 

s  Inferential  -  using  indicators  as  surrogates  for  direct 
measures 

s  Verbal  characterizations  -  e.g.  team  work,  stakeholder 
coordination 


■s  Qualitative  -  cultural  characterizations  about  the  process 
experience  from  implementers  /  users 
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Example  PPQA 
Evolutionary  Measures 

•  QA  milestone  completions 
compared  to  plan 

•  Work  completed,  effort  A 

expended  compared  to  plan 

•  Number  of  product  audits  and 
activities  reviews  compared  to 
plan 

•  Number  of  process  audits  and 
activities  vs.  those  planned 

•  Amount  of  time  /  effort  spent 
on  rework 

•  Amount  of  time  /  effort  spent 
in  each  phase  of  life  cycle 

•  Number  of  defects  per 
release,  build 

•  Total  number  of  defects  found 
by  internal  reviews  and 
verification  activities  vs  those 
found  by  customer  after 
delivery 

•  Number  of  defects  injected  in 
each  phase  of  the  life  cycle  J 

•  Number  of  noncompliance's  / 
nonconformance's  written  vs. 
resolved  vs.  escalated 


For  Maturity  Level  Success 


2 

2 


3 

3 


4 


4 


Variance  of  objective  process 
evaluations  planned  and 
performed 

Variance  of  objective  work 
product  evaluations  planned  and 
performed 

Number  of  process-improvement 

proposals  submitted,  accepted, 
or  implemented 

Defect  density  of  each  process 

element  of  the  organization's  set 
of  standard  processes 

Profile  of  subprocesses  under 
statistical  management  (e.g., 
number  planned  to  be  under 
statistical  management,  number 
currently  being  statistically 
managed,  and  number  that  are 
statistically  stable) 

Number  of  special  causes  of 
variation  identified 
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Generic  Implementation  Interactions 


GP  2.8  Monitor  and 
Control  the  Process 


Day  to  day  sub/process  usage 
&  output  from  a  practitioner  view 

Against  requirements  and 
needs  Goals  /  limits 


Review  /  roll-up  of  sub  process\ 
to  determine  compliance 
\&  improvements 

2.9 

Objectively 
Evaluate 
Adherence 
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2.10  Review 
Status  with 
Higher 
Level 

Management 


Summary  reporting  against  goals 

Progress  against  overall  process 
or  system 

Process  noncompliance 
resolution 


Data  driven  decisions 
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It  is  all  about  data.  And  much  more... 


Misunderstandings 

Any  process  measurement  will 
fill  the  void 

R  Doesn’t  include  qualitative 
expressions  of  process 
monitoring  and  control 

■  Really  only  use  data  at  CL  / 
ML  3  and  above 

■  Information  doesn’t  always 
translate  into  dollars  or  ROI 

■  Is  not  proactive 

■  It  doesn’t  include  mistake 
prevention  or  proofing 

■  Emphasizing  short  run  results 
at  the  expense  of  long-run 
objectives  (myopia) 


Understandings 

■  Start  with  the  end  in  mind  - 
what  do  you  NEED 

Data  comes  from  actual  activity 
or  sub-process  use  rather  than 
generated  CMMI  model 
examples 

Often  use  qualitative  information 
from  “water  cooler”  to  set 
context  for  numbers 

i  Set  the  initial  mechanisms  in 
place  for  eventually  determining 
whether  a  process  is  in  control 
or  out  of  control  (quality  control, 
rework,  etc) 

■  It  is  about  learning  error 
prevention  vs.  just  correction 
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Implementation  Pitfalls 


$  , 


•  «&  ^  ' 


A 
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■  Jumping  right  into  CL/ML  2  without  understanding  the 
processes  or  relationships  among  them  (process  areas  over 
business  processes) 

■  Too  many/few  measures  -  what  isn’t  counted  doesn’t  count 

■  Monitoring  activity  vice  results 

■  Confusion  over  monitoring  process  inputs,  rather  than 
outputs 

■  Data  easily  gathered  rather  than  those  important  for  control 

■  Gold-platting 

s  Using  Earned  Value  on  every  small  /  very  short  during 
projects 

s  Using  production  measures  on  documents 

s  Demanding  detailed  completion  data  and  confusing  it 
with  reality 

l  Difficult  infrastructure  for  reporting 

Misalignment  with  organizational  scorecards  (e.g.  Balanced 
ScoreCard,  metrics  dashboards,  etc.) 

i  Not  a  closed  loop  system  -  collecting  but  not  using 
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Appraisal  Considerations 


l  Stacking  deck  to  misrepresent  or  camouflage  dysfunctional  process, 
project  or  organizational  activities 

Generic  practice  2.8  implementation  across  all  PAs  (CL/ML  L2->L5) 

s  Can  you  show  collection,  usage,  alignment,  limits 
■  Direct  Artifact  Example: 

s  Records  of  evaluations  or  audits  being  performed  as  planned  (e.g., 
reports,  completed  checklists). 

s  Noncompliance  issues  resulting  from  objective  evaluation  of 
adherence  to  processes,  objectives,  and  standards. 

i  Indirect  Artifact  Example: 

s  Revisions  and  change  history  to  plans  and  commitments  (e.g., 
replanned  schedule,  costs,  resources). 

s  Effort  spent  on  the  Process  Area  (e.g.,  reviews  and  action  items 
regarding  activities  and  Process  Area) 

s  Evidence  of  reviews  of  activities,  status,  and  results  of  the  process 
held  with  immediate  level  of  management  responsible  for  the 
process  and  identification  of  issues;  (e.g.  briefings,  reports, 
presentations,  milestones). 

s  Issues  and  corrective  actions  for  deviations  from  plan  for  executing 
the  activities  or  Process  Area  (e.g.,  action  items,  variance  reports, 
change  requests). 
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Contact  Us 


Gary  F.  Norausky 
norausky@norauskypsi.com 
+1(619)  472  8810 

Les  Stamnas 

stamnas@norauskypsi.com 
+1(858)  735  3965 


Norausky  Process  Solutions,  Inc. 

www.norauskypsi.com 
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Topics 


•  Defining  and  Controlling  Production 
Processes 

•  Measuring  Process  Performance 

•  Predicting  Process  Performance 

•  References 
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Definition  of  Process 


A  set  of  activities,  methods,  practices,  and  tools  that 
people  use  to  develop  and  maintain  a  product  and  its 
associated  work  products  (e.g.,  plans,  design 
documents,  code,  test  cases,  and  user  manuals). 
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Types  of  Process  Models 


•  Capability  Maturity  Model 

-  Best  Practices 

-  “Ought  to” 

•  Process  Architecture 

-  Project  life  cycle  model 

-  Activities,  artifacts,  and  timing 

-  High-level  “How  to” 

-  Basis  for  early  planning 

•  Defined  Process 

-  Organization’s  Standard  Process  (OSP) 

-  Detailed  “How  to”  plus  aids  (template  tools) 

•  Project’s  Tailored  Process 

-  Selected  subset  of  the  OSP 

-  Some  elements  may  be  tailored 

-  Basis  for  detailed  planning  (budget,  status)  and  improvement 
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From  Best  Practices  to  Products 


Best 

Practices 


Standard 

Process 


Tailored 

Process 


Instance  of 
the  Tailored 
Process 


Products 
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Process  Control:  Measurements  +  Models 


Legend 

-  Data,  Information,  and  Measurements 

.  Predictions 

- Defined  (and  Improved)  Process 
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Measuring  Process  Performance 


Key  Questions 

-  What  is  the  current  performance? 

-  Is  this  value  "good"? 

-  Is  it  changing? 

-  How  can  I  make  the  value  “better”? 

Candidate  Attributes* 

-  Definition  (completeness,  compatibility) 

-  Usage  (compliance,  consistency) 

-  Stability  (repeatability,  variability) 

-  Effectiveness  (capability) 

-  Efficiency  (productivity,  affordability) 

-  Predictive  Ability  (accuracy,  effects  of  tailoring  and  improvements) 


*Motivated  by  [Florae,  1999,  Section  2.4] 
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Some  Examples 


Goal 

Measure 

Completeness 

The  number  of  process  elements  added,  changed,  and 
deleted  during  tailoring. 

Compliance 

Number  of  discrepancy  reports  generated  by  Quality 
Assurance  audits 

Stability  (volatility) 

The  number  of  process  elements  changed  within  a 
specified  time  interval. 

Effectiveness 

Product  quality 

Effectiveness 

Defect  leakage  to  subsequent  phases 

Efficiency 

Productivity  (or  production  coefficient) 

Efficiency 

Rework  as  a  fraction  of  total  effort 

Predictability 

Probability  distribution  for  an  estimated  quantity  or 
related  population  statistics 

CMMI  User  2005  (14NOV05) 
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Choosing  Your  Measures 


•  Measurement  costs  money 

-  Choose  what  is  useful  (e.g.,  use  Goal  -  Question  -  Measure) 

-  Your  needs  will  change  over  time 

•  Factors  to  consider: 

-  Business  objectives 

-  Customer  desires 

-  Government  regulations  and  statutes 
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Predicting  Process  Performance 


Key  Questions 

How  do  process  parameters  affect  project  productivity,  cost, 
and  schedule? 

How  do  process  parameters  affect  product  quality? 

How  can  I  improve  the  process?  (What  is  the  increase  in 
product  quality  if  I  invest  more  effort  in  design  instead  of 
testing?) 

Process  Performance  Model 

Makes  quantitative  predictions  about  a  particular 
production  process 

May  estimate  resource  consumption,  time  delays, 
effectiveness,  and  efficiency 
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Types  of  Process  Performance  Models 


Type 

Handles  Unstable 
Processes? 

Representation  of 
Process  Mechanisms 

Examples 

Statistical 

No 

None 

Statistical  process  control 

Functional 

No 

Explicit 

Parametric  models  (algorithms 
based  on  causal  mechanisms). 
COQUALMO  and  staged  models. 

Dynamic 

Yes 

Implicit  (via 
propagation) 

System  dynamics  models  (Coupled 
equations  embody  the  causal 
mechanisms.  Solving  numerically 
gives  predicted  behavior.) 

CMMI  User  2005  (14NOV05) 
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Statistical  Process  Control 


CMMI  User  2005  (14NOV05) 
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Sample  Defect  Leakage  Matrix 


Phase  Detected 

Phase  Injected 

Analysis 

Design 

Code 

Integ. 

Test 

Alpha 

Test 

Beta  Test 

Analysis 

98.0 

6.0 

12.0 

14.0 

27.0 

18.0 

Design 

142.0 

38.0 

23.0 

17.0 

8.0 

Code 

114.0 

61.0 

23.0 

4.0 

Integ.  Test 

16.0 

2.0 

1.0 

Alpha  Test 

2.0 

0.0 

Beta  Test 

1.0 
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System  Dynamics  Model:  Concept 
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System  Dynamics  Model: 
Relationships  for  CoSQ 
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System  Dynamics  Model:  Sample 

Output 
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Summary 


•  Measures  help  control  the  production  process 

•  The  choice  of  process  performance  measures 
depends  on  organizational  goals 

•  Predictive  models  supplement  measures 

•  Predictive  accuracy  depends  on 

-  The  process  definition  (detail,  stability,  tailoring) 

-  The  process  execution  (compliance,  consistency) 

-  The  model’s  scope  and  validity  (relevant  factors  and 
interactions,  fidelity) 
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Context  -  CenPRA  -  Centro  de 
Pesquisas  Renato  Archer 


CMMr^S 

Technology 

Conference 

AMP  USERftftPUf 


Brazilian  Government  funded  Research  Center  on  Information 
Technology 

Research  Areas:  Technological  Innovation,  Qualification  and 
Applications  for  the  Society 

230  researchers  in  13  Technological  Divisions 

DMPS:  Divisao  de  Melhoria  de  Processos  de  Software 
Software  Process  Improvement  Division 

•  Focus:  Process  Evaluation  and  Improvement 

•  Activities:  technology  research,  articulation,  dissemination  and 
services 

•  CMM/CMMI,  ISO/IEC  15504  and  MPS.BR  (Brazilian  Software 
Process  Improvement)  models 

•  Participation  in  technical  committees  of  ISO/IEC  15504  and  MPS.BR 

•  Technical  orientation  for  Process  improvement  in  groups  of  small 
enterprises 

•  Training  courses 

•  Specific  Process  Models  development. 
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Motivation 
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•  After  2005  December  no  more  CMM  appraisals 
will  be  held 

•  Organizations  with  CMM  level  2  or  3  have  a 
powerful  tool  for  the  beginning  of  the  transition 
process:  The  independent  verification  of  SQA 
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How  it  began 
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•  In  2004  December,  an  Organization  at  CMM  level  2  has 
asked  us  to  make  an  independent  evaluation  in  order  to 
comply  to  CMM  SQA  Verification  3: 

•  Experts  independent  of  the  SQA  group  periodically  review  the 
activities  and  software  work  products  of  the  project's  SQA  group. 

•  Organization's  Process  Manual  defines  that  the  independent 
evaluation  would  be  conducted  by  external  entity. 

•  Due  to  the  near  deadline  of  CMM  retirement,  we 
proposed  to  the  Organization  to  take  advantage  of  this 
opportunity  to  make  a  more  comprehensive  and  deep 
verification,  making  a  SCAMPI  B  based  assessment  and, 
using  both  CMM-SQA  and  CMMI-PPQA  as  references  for 
the  evaluation. 
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•  The  evaluation  has  followed  the  SCAMPI  phases: 

1 .  Plan  and  Prepare  appraisal 

2.  Conduct  appraisal 

3.  Report  results 


Activities  done  -  Phase  1 : 

Plan  and  Prepare  the  appraisal 


Technology 


ECHNOLOGY 


•  Preliminary  Appraisal  (1  day) 

•  Objectives: 

•  Understand  how  CMM  was  applied  in  the  Organization 

•  SQA  preliminary  appraisal 

•  Define  a  document  list  for  the  appraisal 

•  Present  and  discuss  the  appraisal  format 

•  Meetings  with  the  appraisal  sponsors 

•  Work  with  SQA  people 

•  Initial  collection  of  practice  implementation  indicators  (Pll) 

•  Appraisal  Plan  and  preparation 

•  Appraisal  Plan  was  written  by  CenPRA  based  on  the  Preliminary 
Appraisal,  reviewed  and  complemented 

•  It  defines,  in  compliance  with  the  Organization  Process  Manual: 

•  Scope  and  objectives  of  the  appraisal,  roles  and  responsibilities 

•  Organization's  Departments  to  be  appraised 

•  SQA  process  according  to  SW-CMM  SQA  and  CMMI-SE/SW  PPQA 

•  According  to  SCAMPI-B  (initially  it  was  defined  to  be  C,  but  during  the 
work,  it  was  found  that  the  assessment  could  be  level  B) 
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Activities  done  -  Phase  2: 
Conducting  the  appraisal  (1,5  day) 


•  Appraisal  done  according  to  the  Plan: 

•  Organization’s  appraiser  training 

•  Opening  presentation  for  the  participants 

•  Documents  verification 

•  Interviews:  project  teams,  project  managers,  process 
improvement  groups,  SQA 

•  Results  compilation 
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Activities  done  -  Phase  3: 
Results  Reporting 
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•  Results  presented  to  sponsors  and  appraisal  participants. 

•  SQA  Process  is  implemented  and  institutionalized  compliant  to  SQA 
KPA  and  PPQA  PA;  none  non-conformity  was  identified. 

•  SQA  provides  senior  management  visibility  of  inconsistencies, 
supports  the  execution  and  improvement  of  the  processes,  and 
helps  maintaining  the  compliance  of  all  processes  to  SW-CMM  level 
2. 

•  In  the  transition  to  CMMI,  it  will  be  necessary  to  elaborate  new 
instruments  for  the  verification  of  the  PA’s,  but  the  SQA  process 
itself  would  be  kept  as  is  because  it  is  compliant  to  PPQA 

•  The  SQA  role  is  perceived  as  a  reference  and  support  for  each 
one’s  job  to  comply  with  SW-CMM 

•  The  SQA  Status  Report  presents  in  a  clear  and  concise  form,  the 
results  of  monthly  evaluations,  and  the  corrective  actions 

•  There  is  a  process  and  practice  for  process  revision  and 
improvement 
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Conclusions  of  the  case  study 
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•  The  existence  of  a  very  good  “SQA  Status 
Report”  contributes  to  a  comprehensive  and  deep 
insight 

•  Assessing  the  level  2  KPA’s  based  in  this  Report 
is  a  simple  matter  of  confirmation  of  the 
verification  of  the  Process  Implementation 
Indicators  that  has  been  verified  previously  by  the 
organization's  SQA  function 

•  It’s  possible  to  use  the  independent 
verification  as  a  first  step  in  the  transition 
from  SW-CMM  to  CMMI-SE/SW 
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•  The  more  the  SQA  function  is  well  done,  the  more  it’s  possible 
to  use  the  SQA  independent  verification  to  identify  the  gaps  to 
CMMI-SE/SW 

•  In  the  preparation  phase,  it  is  possible  to  have  a  first  insight  of 
how  good  is  the  SQA  and  if  it  is  possible  to  apply  the  proposed 
approach 

•  In  this  case  study: 

•  The  "SQA  Status  Report”  reports  the  non-conformities  for  each 
KPA  against  the  Goals,  Commitment,  Abilities,  Activities, 
Measurements,  and  Verification.  Based  on  this,  the  Process 
Implementation  Indicators  can  be  identified. 

•  It’s  possible  to  extract  the  frequency,  recurrence  and  trends  of  the 
non-conformities,  allowing  a  priority  analysis  of  the  problems. 

•  It  is  a  very  good  instrument  for  the  appraisal  of  the  Level  2  PAs 

•  If,  as  in  this  case  study,  it  is  perceived  that  it  would  be  possible 
to  extend  the  objectives  of  the  independent  verification  to  a  first 
gap  analysis,  make  the  plan  for  doing  so 
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•  Define,  with  the  appraisal  sponsor,  the  intention  to  use  the 
SQA  independent  appraisal  for  the  gap  analysis 

•  Verify  the  availability  of  SQA  instruments  like  that  "SQA 
Status  Report". 

•  If  it  has  been  concluded  that  there  is  good  information,  do 
the  following  activities: 

a)  Include  the  goal  of  identifying  the  gaps  to  CMMI  in  the  Appraisal  Plan 

b)  Use  a  SW-CMM  KPA's  to  CMMI  PA's  mapping 

c)  From  a  “SQA  Status  Report”  or  equivalent,  raise  the  objective  evidences 
not  only  for  SQA/PPQA  but  also  for  the  other  PA’s. 

d)  Identify  other  Process  Implementation  Indicators  (Pll)  for  other  PA’s 

e)  Adapt  the  Appraisal  Plan,  the  agenda,  required  resources  and  documents 
to  the  additional  goal  of  identifying  the  gaps 

f)  In  the  team  training,  include  some  comparison  between  CMM  and  CMMI 
highlighting  the  differences  and  similarities.  Include  also  some 
comparison  between  SCAMPI  and  CBA-IPI. 
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What  to  do 

2  -  Conducting  the  appraisal 
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•  Conduct  the  appraisal  according  to  SCAMPI 
(B  or  C)  verifying  the  SQA  function  and  the 
processes  verified  by  it 

•  Organize  the  results  separating  the  CMM  issues 
and  the  CMMI  issues 
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What  to  do 

3  -  Reporting  the  results 
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•  The  report  will  be  done  in  two  distinct  parts: 

•  CMM-SQA  issues 

•  CMMI  gaps  issues 

•  Take  advantage  of  the  closing  session  to  explain 
differences  between  CMM  and  CMMI  and,  CBA- 
IPI  and  SCAMPI 
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•  First  gap  analysis  with  small  increase  in  the 
appraisal  costs  -  it  can  be  used  as  input  for  the 
decision  about  how  and  when  the  transition  will 
be  done 

•  The  Organization  will  have  a  contact  with  CMMI 
and  SCAMPI,  in  a  practical  way,  through  the 
opening  and  closing  sessions,  and  with  the 
objective  evidences  gathering  and  the  interviews 

•  SQA  people  can  reformulate  their  instruments  for 
their  internal  evaluations  for  a  better  transition 
support 
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•  This  work  is  an  application  of  the  “Industry  as  Laboratory” 
concept  proposed  by  Potts  [5].  The  initial  intent  of  the  industry 
aimed  to  a  specific  necessity:  to  perform  independent 
verification  of  SW-CMM  SQA.  We,  from  CenPRA,  proposed 
to  widen  the  appraisal  using  CMMI  PPQA  as  reference  and 
SCAMPI  B  as  the  appraisal  method. 

•  The  results  of  the  appraisal  indicate  us  the  possibility  of 
proposing  an  even  more  widening  of  the  appraisal  objectives: 
use  the  independent  verification  of  SW-CMM  SQA  for  the  gap 
identification  for  the  transition  from  SW-CMM  to  CMMI- 
SE/SW 

•  We  have  defined  “What  to  do”  to  apply  this  approach 

•  We  intend  to  apply  this  approach  soon  in  the  same 
organization  or  in  another  one 
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Context  -  CenPRA  -  Centro  de 
Pesquisas  Renato  Archer 


CMMr^S 
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Brazilian  Government  funded  Research  Center  on  Information 
Technology 

Research  Areas:  Technological  Innovation,  Qualification  and 
Applications  for  the  Society 

230  researchers  in  13  Technological  Divisions 

DMPS:  Divisao  de  Melhoria  de  Processos  de  Software 
Software  Process  Improvement  Division 

•  Focus:  Process  Evaluation  and  Improvement 

•  Activities:  technology  research,  articulation,  dissemination  and 
services 

•  CMM/CMMI,  ISO/IEC  15504  and  MPS.BR  (Brazilian  Software 
Process  Improvement)  models 

•  Participation  in  technical  committees  of  ISO/IEC  15504  and  MPS.BR 

•  Technical  orientation  for  Process  improvement  in  groups  of  small 
enterprises 

•  Training  courses 

•  Specific  Process  Models  development. 
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Motivation 
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•  After  2005  December  no  more  CMM  appraisals 
will  be  held 

•  Organizations  with  CMM  level  2  or  3  have  a 
powerful  tool  for  the  beginning  of  the  transition 
process:  The  independent  verification  of  SQA 
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How  it  began 
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•  In  2004  December,  an  Organization  at  CMM  level  2  has 
asked  us  to  make  an  independent  evaluation  in  order  to 
comply  to  CMM  SQA  Verification  3: 

•  Experts  independent  of  the  SQA  group  periodically  review  the 
activities  and  software  work  products  of  the  project's  SQA  group. 

•  Organization's  Process  Manual  defines  that  the  independent 
evaluation  would  be  conducted  by  external  entity. 

•  Due  to  the  near  deadline  of  CMM  retirement,  we 
proposed  to  the  Organization  to  take  advantage  of  this 
opportunity  to  make  a  more  comprehensive  and  deep 
verification,  making  a  SCAMPI  B  based  assessment  and, 
using  both  CMM-SQA  and  CMMI-PPQA  as  references  for 
the  evaluation. 
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•  The  evaluation  has  followed  the  SCAMPI  phases: 

1 .  Plan  and  Prepare  appraisal 

2.  Conduct  appraisal 

3.  Report  results 


Activities  done  -  Phase  1 : 

Plan  and  Prepare  the  appraisal 


Technology 


ECHNOLOGY 


•  Preliminary  Appraisal  (1  day) 

•  Objectives: 

•  Understand  how  CMM  was  applied  in  the  Organization 

•  SQA  preliminary  appraisal 

•  Define  a  document  list  for  the  appraisal 

•  Present  and  discuss  the  appraisal  format 

•  Meetings  with  the  appraisal  sponsors 

•  Work  with  SQA  people 

•  Initial  collection  of  practice  implementation  indicators  (Pll) 

•  Appraisal  Plan  and  preparation 

•  Appraisal  Plan  was  written  by  CenPRA  based  on  the  Preliminary 
Appraisal,  reviewed  and  complemented 

•  It  defines,  in  compliance  with  the  Organization  Process  Manual: 

•  Scope  and  objectives  of  the  appraisal,  roles  and  responsibilities 

•  Organization's  Departments  to  be  appraised 

•  SQA  process  according  to  SW-CMM  SQA  and  CMMI-SE/SW  PPQA 

•  According  to  SCAMPI-B  (initially  it  was  defined  to  be  C,  but  during  the 
work,  it  was  found  that  the  assessment  could  be  level  B) 
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Activities  done  -  Phase  2: 
Conducting  the  appraisal  (1,5  day) 


•  Appraisal  done  according  to  the  Plan: 

•  Organization’s  appraiser  training 

•  Opening  presentation  for  the  participants 

•  Documents  verification 

•  Interviews:  project  teams,  project  managers,  process 
improvement  groups,  SQA 

•  Results  compilation 
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Activities  done  -  Phase  3: 
Results  Reporting 
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•  Results  presented  to  sponsors  and  appraisal  participants. 

•  SQA  Process  is  implemented  and  institutionalized  compliant  to  SQA 
KPA  and  PPQA  PA;  none  non-conformity  was  identified. 

•  SQA  provides  senior  management  visibility  of  inconsistencies, 
supports  the  execution  and  improvement  of  the  processes,  and 
helps  maintaining  the  compliance  of  all  processes  to  SW-CMM  level 
2. 

•  In  the  transition  to  CMMI,  it  will  be  necessary  to  elaborate  new 
instruments  for  the  verification  of  the  PA’s,  but  the  SQA  process 
itself  would  be  kept  as  is  because  it  is  compliant  to  PPQA 

•  The  SQA  role  is  perceived  as  a  reference  and  support  for  each 
one’s  job  to  comply  with  SW-CMM 

•  The  SQA  Status  Report  presents  in  a  clear  and  concise  form,  the 
results  of  monthly  evaluations,  and  the  corrective  actions 

•  There  is  a  process  and  practice  for  process  revision  and 
improvement 
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•  The  existence  of  a  very  good  “SQA  Status 
Report”  contributes  to  a  comprehensive  and  deep 
insight 

•  Assessing  the  level  2  KPA’s  based  in  this  Report 
is  a  simple  matter  of  confirmation  of  the 
verification  of  the  Process  Implementation 
Indicators  that  has  been  verified  previously  by  the 
organization's  SQA  function 

•  It’s  possible  to  use  the  independent 
verification  as  a  first  step  in  the  transition 
from  SW-CMM  to  CMMI-SE/SW 


CenPRA 

Centro  de  Pesquisas 
Renato  Archer 


What  to  do 
0  -  Prerequisite 


CMMT^S 

Technology 

CONFfRiNCE 

USEttftRPUf 


•  The  more  the  SQA  function  is  well  done,  the  more  it’s  possible 
to  use  the  SQA  independent  verification  to  identify  the  gaps  to 
CMMI-SE/SW 

•  In  the  preparation  phase,  it  is  possible  to  have  a  first  insight  of 
how  good  is  the  SQA  and  if  it  is  possible  to  apply  the  proposed 
approach 

•  In  this  case  study: 

•  The  "SQA  Status  Report”  reports  the  non-conformities  for  each 
KPA  against  the  Goals,  Commitment,  Abilities,  Activities, 
Measurements,  and  Verification.  Based  on  this,  the  Process 
Implementation  Indicators  can  be  identified. 

•  It’s  possible  to  extract  the  frequency,  recurrence  and  trends  of  the 
non-conformities,  allowing  a  priority  analysis  of  the  problems. 

•  It  is  a  very  good  instrument  for  the  appraisal  of  the  Level  2  PAs 

•  If,  as  in  this  case  study,  it  is  perceived  that  it  would  be  possible 
to  extend  the  objectives  of  the  independent  verification  to  a  first 
gap  analysis,  make  the  plan  for  doing  so 
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CenPRA 

Centro  de  Pesquisas 
Renato  Archer 


What  to  do 
1-  Preparation 


CMMT^S 

Technology 

CONFERENCE 


•  Define,  with  the  appraisal  sponsor,  the  intention  to  use  the 
SQA  independent  appraisal  for  the  gap  analysis 

•  Verify  the  availability  of  SQA  instruments  like  that  "SQA 
Status  Report". 

•  If  it  has  been  concluded  that  there  is  good  information,  do 
the  following  activities: 

a)  Include  the  goal  of  identifying  the  gaps  to  CMMI  in  the  Appraisal  Plan 

b)  Use  a  SW-CMM  KPA's  to  CMMI  PA's  mapping 

c)  From  a  “SQA  Status  Report”  or  equivalent,  raise  the  objective  evidences 
not  only  for  SQA/PPQA  but  also  for  the  other  PA’s. 

d)  Identify  other  Process  Implementation  Indicators  (Pll)  for  other  PA’s 

e)  Adapt  the  Appraisal  Plan,  the  agenda,  required  resources  and  documents 
to  the  additional  goal  of  identifying  the  gaps 

f)  In  the  team  training,  include  some  comparison  between  CMM  and  CMMI 
highlighting  the  differences  and  similarities.  Include  also  some 
comparison  between  SCAMPI  and  CBA-IPI. 
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Renato  Archer 


What  to  do 

2  -  Conducting  the  appraisal 


CMMT^S 

Technology 

CONFERENCE 


•  Conduct  the  appraisal  according  to  SCAMPI 
(B  or  C)  verifying  the  SQA  function  and  the 
processes  verified  by  it 

•  Organize  the  results  separating  the  CMM  issues 
and  the  CMMI  issues 
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CenPRA 

Centro  de  Pesquisas 
Renato  Archer 


What  to  do 

3  -  Reporting  the  results 


CMMT^S 

Technology 

CONFERENCE 

ANP  USER  ft  M3  UP 


•  The  report  will  be  done  in  two  distinct  parts: 

•  CMM-SQA  issues 

•  CMMI  gaps  issues 

•  Take  advantage  of  the  closing  session  to  explain 
differences  between  CMM  and  CMMI  and,  CBA- 
IPI  and  SCAMPI 
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CenPRA 

Centro  de  Pesquises 
Renato  Archer 


SQA  independent  verification  used  as 
first  step  for  the  transition  to  CMMI 


CMMT^S 

Technology 

Conference 
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Advantages 


CMMT^S 

Technology 

CONFERENCE 


•  First  gap  analysis  with  small  increase  in  the 
appraisal  costs  -  it  can  be  used  as  input  for  the 
decision  about  how  and  when  the  transition  will 
be  done 

•  The  Organization  will  have  a  contact  with  CMMI 
and  SCAMPI,  in  a  practical  way,  through  the 
opening  and  closing  sessions,  and  with  the 
objective  evidences  gathering  and  the  interviews 

•  SQA  people  can  reformulate  their  instruments  for 
their  internal  evaluations  for  a  better  transition 
support 
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CenPRA 

Centro  de  Pesquisas 
Renato  Archer 


Conclusions 


CMMT^S 

Technology 

CONFfRiNCE 

^HP  USittftRPUf 


•  This  work  is  an  application  of  the  “Industry  as  Laboratory” 
concept  proposed  by  Potts  [5].  The  initial  intent  of  the  industry 
aimed  to  a  specific  necessity:  to  perform  independent 
verification  of  SW-CMM  SQA.  We,  from  CenPRA,  proposed 
to  widen  the  appraisal  using  CMMI  PPQA  as  reference  and 
SCAMPI  B  as  the  appraisal  method. 

•  The  results  of  the  appraisal  indicate  us  the  possibility  of 
proposing  an  even  more  widening  of  the  appraisal  objectives: 
use  the  independent  verification  of  SW-CMM  SQA  for  the  gap 
identification  for  the  transition  from  SW-CMM  to  CMMI- 
SE/SW 

•  We  have  defined  “What  to  do”  to  apply  this  approach 

•  We  intend  to  apply  this  approach  soon  in  the  same 
organization  or  in  another  one 
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Agenda 


■  Our  Process  Improvement  History 

■  The  Infrastructure  That  Made  It  Work 

■  New  Attitudes  In  Using  Metrics 

■  Is  Level  5  The  End  ...  Or  The  Beginning 
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Northrop  Grumman  Today 


•  125,000  people,  50  states,  25  countries 

•  Largest  manufacturing  employer  in 
Louisiana,  Mississippi,  Virginia,  Maryland 

•  One  of  top  three  defense  contractors 

•  Leading  systems  integrator 

•  Largest  military  shipbuilder 

•  Largest  provider  of  airborne  radar  and 
electronic  warfare  systems 

•  One  of  two  top  IT  providers  to  the  U.S. 
Government 

•  One  of  three  major  contractors  in  military 
and  civil  space,  missile  defense 


More  than  $31  Billion 


in  2004  Sales 


NORTHROP  GRUMMAN 


afT 


AGS&BMS-PR-05-1 03 
•\  IfTftP  P 


Return  on  Investment  Demonstrated  by 
•Higher  Productivity  •Improved  Quality 
•Competitive  Pricing  •Faster  Time  to  Market 
Since  Our  Work  Is  Primarily  Cost-Plus,  These 
Benefits  Accrue  to  Our  Customers 


992  1995  1998  2001  2004 
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Infrastructure  for  Innovation 


Corporation  & 
Business  Area 


Corporate  goals 
Business  Area 
goals 


Direction, 
Guidance 
Resources 
Engineering 
goals,  objectives 


Engineering 
Process  Group 
(EPG) 


Software  Engineering 
Process  Group  (SEPG) 


Software  practitioners 
and  relevant 
stakeholders  to 
improve  software 
specific  processes 


Status  reporting 


•  Engineering  Directors 

•  Quality  Director 

•  Executive  Management  Representative 

Improvement 

proposals 

Process  performance 
status  reporting 

•  Full  Time  EPG 
Chairperson 

•  Representatives  from 
each  Engineering 
Directorate 


Software  Quality 
representative 
Program  and  project 
representatives 


Tocess  Management 
Teams  (PMT) 

Multi-disciplinary  teams 
empowered  to  evaluate 
and  continuously  improve 
broad  engineering 
processes 


Teams  established  as 
needed 

NORTHROP  GRUMMAN 
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Steering  Committee 


■  Comprises 

■  Engineering  Director 

■  Directors  from  Each  Engineering  Directorate 
(Systems,  Software,  Test,  Vehicle,  Avionics,  Logistics) 

■  Quality  Operations 

■  Business  Area  Management  Rep 

■  Project  Engineering  Managers 

■  Program  Managers 

■  Engineering  Process  Group 

■  Meets  Every  Week  to  Review  Process  Improvement 
Status  with  EPG  and  Project  Practitioners 

■  Government  Reps  Invited  to  Meetings 


MWIWHOP  GRUMMAN 


Engineering  Process  Group  (EPG) 


■  Made  Up  of  Process  Definition  and  Management 
Personnel  in  Each  Engineering  Directorate 

■  Facilitates  Process  Improvement  across  the 
Engineering  Department 

■  Maintains  Process  Assets  for  Use  by  the 
Organization 

■  Coordinates  with  Organizations  Outside  of 
Engineering  to  Ensure  Proper  and  Efficient  Process 
Interfaces 

■  Facilitates  Compliance  with  Appropriate  Process 
Standards  and  Models  (E.G.,  ISO  9001 ,  CMMI) 

■  Manages  Engineering  Process  Management  Teams 

■  Develops  and  Maintains  Relationships  with 

Universities,  Research  Labs  and  Related  Consortia  to 
Support  Engineering  Goals  NORTHROP  GRUMMAN 
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Process  Management  Teams 

Focusing  Lean  on  Significant  Issues 


Facilitators 


r 

Engineering  PMT 

Steering  Committee 

A 

Software 

Engineering 


Systems 
Engineering 
Life  Cycles 


COTS  and 
PME 


Logistics 

Commodities 


Test  and 
Integration 


ILS 

Processes 


System 

Integration 

Labs 


Vehicle 

Engineering 
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Engineering  PMTs  -  General  Goals 


■  Map  Process  Value  Stream  for  the  Production  of 
Relevant  Products 

■  Determine  Non-Value  Added  Activities 

■  Recognize  That  Some  of  These  May  Be  Required  by 
Customers  or  Business  Needs 

■  Identify  Issues  or  Concerns  Regarding  the  Process  or 
Product 

■  Execute  Causal  Analysis  &  Resolution  Process  As 
Needed 

■  Determine  Alternatives  to  the  Current  Way  of  Doing 
Business 

■  Propose  “Best”  Alternatives  in  Terms  of  Cost, 

Schedule,  Quality  or  Productivity  Improvements 

■  Present  Alternatives  to  Steering  Committee  for 

Selection  for  Implementation  NORTHROP  GRUMMAN 
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CMMI  Higher  Levels  — 
Differences  in  Behavior 


At  Level  3 . 

At  Level  4 . 

•  Management  Reacts 

•  Management  Anticipates 

•  Comparative  Rather  Than 

•  Predicting  Results  of 

Statistical  Analysis 
•  Process  Capability  Not 

Critical  Processes 
•  Evaluating  Outcomes 

Understood 

Relative  to  Capability 

•  Measurement  Program 

•  Measurement  Program 

•  Data  Available  for 

•  Data  Relied  on  for 

Analysis 

Decision-making 

•  Analysis  at  Project  Level 

•  Data  Analyzed  at 

•  Data  Quality  Often  Still  a 

Organization  and  Project 

Concern 

Levels 

At  Level  5 . 

•  Management  Performs 
“Pre-emptive  Strikes” 

•  Identifying  &  Removing 
Systemic  Process  Issues 

•  Predicting  Results  of 
Innovative  Improvements 


Measurement  Program 

•  Data  Relied  on  for 
Cost/Benefit  Analysis 

•  Benefits  Forecasted 
for  Technology  or 
Process  Optimization 


NORTHROP  GRUMMAN 
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Using  Metrics  for  Higher  Maturity 


■  Estimating 

■  Base  Estimates  Of  Future  Performance  On  Past 
Performance 

■  Project  Planning 

■  Determine  Resources  Needed  For  Project  Execution 

■  Project  Tracking 

■  Determine  Whether  Actual  Performance  Matches 
Predictions 


Quantitative  Management  Higher  Maturity  Uses  of  Metrics^ 

■  Determine  Whether  Project  Objectives  Are  Likely  To 
Be  Met 

■  Process  Improvement 

■  Determine  Whether  Process  Changes  Flave  Improved 
Performance 


NORTHROP  GRUMMAN 
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Voice  of  the  Process 

Quantitative  Sub-Process  Management 


ASU  Log  Cost  Model 

Using  Lognormal  Probability  Density  Function 


5  ■ 

>4 


Review  Closed  Date 


Upper  Control  Limit 


Average  performance 


Lower  Control  Limit 


■  A  Stable  Process 

■  Operates  Within  the  Control  Limits  99.7%  of  the  Time 

■  Meets  Budget 

■  Offers  Opportunities  for  Systematic  Process  Improvement 

NORTHROP  GRUMMAN 
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Improving  the  Process 


Peer  Reviews  Greater 
Than  1  Standard  Deviation 
Above  the  Average  of  Peer 
Review  Performance 


Peer  Reviews  Greater 
Than  1  Standard 
Deviation  Below  the 
Average  of  Peer  Review 
Performance 


Question:  Is  There  a  Common  Cause  for  the  Variation  in  Either  of 
These  Two  Sub-populations  of  the  Peer  Review  Data? 


MWIWHOP  GRUMMAN 


Develop  Candidate  Solutions  (Example) 


Proposed  Solution 

Comments  for  Evaluation 

Count  the  actual  code  reviewed 
(vs.  just  new  or  modified  code) 

This  is  a  potential  BOE  issue  and  needs  criteria 
for  setting  boundaries  for  code  to  be  reviewed 

Increase  the  complexity  factor  for 
“ 

For  2  or  less  SLOC/unit  set  complexity  to  “1 0”. 

For  other  small  reviews  this  may  need  a 
“calibration  chart”  to  determine  appropriate 
complexity  factors 

For  small  reviews,  select  a 
different  verification  method 

The  different  verification  method  will  need 
definition.  Q:  Are  these  all  Engineering  Checks? 
More  analysis  may  be  needed. 

Automate  the  administrative  work 
Required  to  set  up  peer  reviews 
(e.g.,  create  diff  files,  place  files 
into  a  directory/CMS,  .  .  . ) 

This  change  would  impact  all  reviews  -  not  just 
the  sub-population.  Need  to  evaluate  the  impact 
to  the  overall  population 

NORTHROP  GRUMMAN 


Improvement  in  Process  Performance 


Now 

Current  performance 


time 


March  ‘05  - 
Improved  performance  ^ 


> 


*  At *  * - 

We  Need  To  Minimize  This  Time: 

•  Identify  Improvement  Proposals 

•  Evaluate  &  Prioritize  Proposals 

•  Select  Improvement 

•  Pilot  Improvement 

•  Deploy  Improvement 

MORTbiROP  GRUMMAN 
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Deploying  Improvements 


Publish  a  New  Organization  Baseline  for  the 
Improved  Process 

Deploy  New  Process  Objectives  To  Project 

Deploy  New  Process  To  Project 

Monitor  New  Process  Performance  Against  New 
Capability 


Old  Performance! 


Growing  the  Capability 


What  happens  after  Level  5 .. . 


CMMI  ‘Volume 
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AV  Disc/Inspection  Rpt 
SW  Designs 

ve  Drawings  More  Optimized 

SE  System  Integration  Labs  Processes 
LOG  AFTOs 
TE  Test  Plans/Reports 
SW  Code  Reviews 


Software 

Systems 


^  ^  Logistics 

^  ^  A  Vehicle 

v*  V  K?  fjt  <£  Avionics 

Prog  Mgmt,  Materiel 
O 


More 
Disciplines 


More  Projects 
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QUESTIONS 


I 


Joseph  V.  Vandeville 

Northrop  Grumman  Corporation 
(321)  951-5287 
joseph.vandeville@ngc.com 


Richard  L.  W.  Welch,  PhD 

Northrop  Grumman  Corporation 
(321)  951-5072 
rick.welch@ngc.com 
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Raytheon 

Customer  Success  Is  Our  Mission 


Creating  Helpful  Process 
Directives 

Ken  Weinberg 

El  Segundo,  CA 

ki we  i  n  be  rg@  ray  t  h  eo  n .  co  m 


November  16,  2004 


Raytheon 

Three  Facets  to  Effective  Directives 

•  Directive  System  Architecture 

•  Structure  of  Directives  within  Architecture 

•  Writing  Style  of  Each  Directive 


Each  part  compliments  the  other 


Page  2 


One  Set  of  Directives 


Raytheon 


•  Medium/Large  Programs 

•  Small  Projects 

•  Research  Programs 

•  Engineering  Services 


Goal:  Develop  a  Single  Directive  System  Scalable  to 
Accommodate  Diverse  Types  of  Typical  Programs 
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Directive  System  Architecture 


Directive/  High  Level 

Non-Tailorable  Directly  Traceable 

to  CMMI,  ISO,  Corp  Stds 


Work 

Instructions 


Enablers 


Lower  Level, 

Directive/  Further  Direction  on 
Tailorable  “How”  to 

Meet  Requirement 


Non-Directive 


Guidelines/ 

Templates 


Project  and  Organization  level  Directives  are  Separate 


Page  4 


Directive  Structure 


1 .  Document  Information 

2.  Interfaces 

3.  Inputs/Outputs 

4.  Definitions 

5.  Instruction 

6.  Requirements 

7.  Revision  History 


Raytheon 


Document  Information 


Raytheon 


•  Administrative  -  Document  Number,  Date,  Revision 

•  Summary  -  No  more  than  two  sentences 

•  Intended  Users 

•  Stakeholders 

•  Interfaces  (Optional  in  Enablers) 

-  Identify  Referenced  Documents  by  Document  Number,  Document  Title,  and 
Directive  Type 

•  Inputs/Outputs  (Optional  in  Enablers) 

-  Inputs  -  Any  conditions,  materials,  requirements,  or  outputs  of  other  processes 
necessary  to  begin  the  process 

-  Outputs  -  all  outputs  of  the  process  to  which  the  document  relates,  including 
deliverable  products,  or  products  that  require  storage  in  a  project  or 
organizational  database  or  repository 

•  Definitions  (Optional  in  Enablers) 

-  Terms  specific  to  the  organization  or  the  directive  system;  and  terms  whose 
meaning  differs  from  the  common  dictionary  meaning. 

-  Hyperlink  to  its  Definition  in  the  Glossary 
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Instruction 


Raytheon 


•  Provide  a  narrative  description  that  tells  users  how  to 
execute  the  process. 

-  Amplifies  the  Requirements 

-  Adds  Explanatory  or  Tutorial  Information 

-  Places  the  Process  and  its  Requirements  in  a  Coherent  Narrative 

•  Identify  requirements  in  bold  type. 

•  Keep  the  directives  as  concise  as  possible. 

•  Where  possible,  make  the  process  scalable  to  account  for 
projects  of  different  sizes  and  types. 
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Requirements 


Raytheon 


•  Table  That  Lists  Each  Requirement 

-  Core  of  a  Procedure  or  Work  Instruction 

-  Not  part  of  enablers 

-  Assigns  a  Unique  Number  to  Each  Requirement 

-  Maps  Each  Requirement  to  a  Narrative  Paragraph 

•  Programs  and  Organization  are  Responsible  Only  for 
Complying  with  the  Requirements  in  the  Table  (As  Tailored) 

•  Table  Used  as  Input  for  Tailoring  Tool 

•  Objective  Evaluation  Checklists  Derived  from  Table 
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Writing  Style 
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•  Document  naming  conventions 

•  Discipline  specific  Directives  begin  with  the  discipline 
designation  (i.e.,  “Program  Management”,  “PM”,  “Software”, 
“SW”,  etc.) 

•  Spell  out  the  first  use  of  acronyms  and  abbreviations 

•  Refer  to  directive  documents  by  hyperlinked  document 
number,  italicized  title,  and  directive  type 
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Writing  Style  -  continued 


Raytheon 


•  The  first  sentence  of  each  paragraph  identifies  the 
responsibility  for  that  paragraph 

•  active  voice,  present  tense,  indicative  mood— “who  does 
what.” 

•  Identify  by  role  the  responsible  person  or  team;  do  not  use 
“the  project”  or  “the  organization.”  Do  not  state  that 
someone  “ensures”  or  “assures”  that  something  happens 
unless  the  requirement  is  strictly  a  verification  function. 

•  If  the  same  people  are  responsible  for  the  entire  document, 
identify  the  responsibility  in  the  first  paragraph;  write  the 
rest  of  the  document  in  “directive”  style. 
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Writing  Style  -  continued 


Raytheon 


•  Directive 

-  Subsequent  sentences  in  the  paragraph  provide  directive  instructions. 
Use  the  active  voice,  present  tense,  imperative  mood  -  begin  with  an 
action  verb. 

-  Examples:  “Prepare  the  charts  using  the  format  in  Appendix  B.” 
“Submit  comment  sheets  to  the  review  scribe.”  “Update  the  sizing 
estimates  monthly.” 

•  Explanatory 

-  If  any  further  information  is  needed  to  explain  a  directive,  use  the 
active  voice,  present  tense,  indicative  mood. 

-  Example:  “Sizing  includes  measured  and  projected  usage.” 
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Avoid 


Raytheon 


Passive  voice.  Instruct  the  reader  to  “do  something”  rather  than  stating  that  “something  is  done.” 

Example:  “The  technical  lead  reviews  the  documents.  The  reviewers  each  record  their  comments  on  the  comment 
sheets.  The  review  team  assigns  action  items.” 

Rather  than  “Documents  are  reviewed,  comments  are  recorded  on  comment  sheets,  and  corrective  action  is 
assigned.” 

“perform”  Use  an  imperative  verb  that  defines  what  to  do. 

Example:  “Tailor”  instead  of  “Perform  tailoring  activities.”;  “ 

Objectively  evaluate  the  process”  instead  of  “Perform  objective  evaluation  of  the  process.” 

“the  following.”  Omit  or  use  an  imperative  verb.  Example:  “Consider:”  instead  of  “Consider  the 
following.” 

“in  accordance  with.”  Use  “according  to”  to  specify  a  directive  document  with  which  a  requirement 
complies.  Example:  “Analyze  the  data  according  to  El-98-45, 

“ personnel .”  Identify  the  actual  role.  Example  “The  Program  Configuration  Management  representative 
controls  the  work  product  list”  instead  of  “Configuration  Management  personnel  control  the  work 
product  list.” 

“in  order  to.”  Use  “to”.  Example:  “Reviewers  complete  comment  sheets  to  record  their  comments.” 
instead  of  “Reviewers  complete  comment  sheets  in  order  to  record  their  comments.” 

“ prior  to”  (or  “prior3’).  Use  “before,”  which  means  the  same  thing. 

“utilize.”  Use  “use.” 

“On  a  ...  basis.”  Specify  the  time  interval.  Examples:  “Publish  the  report  monthly”  instead  of  “Publish 
the  report  on  a  monthly  basis.” 

“activity’  when  referring  to  an  individual,  team,  or  organizational  entity.  Specify  the  entity. 
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Summary 
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•  Goals 

-  ISO/AS9100,  Corporate  Standards,  CMMI  model  compliant,  as  scoped 

-  Single,  user-friendly  directive  system 

•  Method 

-  Use  generic  wording  where  possible 

Create  a  Facilities  Plan  Document  Facility  Planning 
SOW  Tasks 

-  Keep  it  short  and  simple  really  short  and  simple 

-  Rely  heavily  on  non-directive  templates  and  guidelines 

•  Three  Facets  to  Effective  Directives 

-  Directive  System  Architecture 

-  Structure  of  Directives  within  Architecture 

-  Writing  Style  of  Each  Directive 
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Questions  ?  ?  ? 
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Topics 


■  Business  Objectives 

■  CMMI  Requirements  for  Sub-process  Control 

■  Why  Peer  Reviews? 

■  Data  Characteristics  and  Difficulties 

■  Log-Return  Model  /  Log-Cost  Model 

■  The  Lognormal  Distribution 

■  Our  Code  Walkthrough  Data  on  Logs 

■  Expanding  the  Capability 

■  Summary 
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CMMI  Higher  Levels  — 
Differences  in  Behavior 


At  Level  3 . 

•  Management  Reacts 

•  Comparative  Rather  Than 
Statistical  Analysis 

•  Process  Capability  Not 
Understood 

•  Measurement  Program 

•  Data  Available  for 
Analysis 

•  Analysis  at  Project  Level 

•  Data  Quality  Often  Still  a 
Concern 


At  Level  4 . 

•  Management  Anticipates 

•  Predicting  Results  of 
Critical  Processes 

•  Evaluating  Outcomes 
Relative  to  Capability 

•  Measurement  Program 

•  Data  Relied  on  for 
Decision-making 

•  Data  Analyzed  at 
Organization  and  Project 
Levels 


At  Level  5 . 

•  Management  Performs 
“Pre-emptive  Strikes” 

•  Identifying  &  Removing 
Systemic  Process  Issues 

•  Predicting  Results  of 
Innovative  Improvements 


Measurement  Program 

•  Data  Relied  on  for 
Cost/Benefit  Analysis 

•  Benefits  Forecasted 
for  Technology  or 
Process  Optimization 

NORTHROP  GRUMMAN 
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Quantitative  Management 

CM  MI  Level  4 

■  Establish  an  Organizational  Baseline  and  Models  of 
Process  Performance 

■  Average  Performance  (Effort,  Duration,  Quality,  ...) 

■  Range  of  Performance  Variation 

■  Contribution  of  Sub-process  Performance  to  Higher 
Level  Processes 

■  Manage  Project  To  Achieve  Quantitative  Process 
Performance  Goals 

■  Establish  Project  Goals  Based  on  Organizational 
Performance 

■  Select  Sub-processes  To  Quantitatively  Manage 

■  Demonstrate  Quantitative  Control 

■  Identify  and  Correct  Special  Causes  of  Performance 
Variation 

■  Feed  Data  Back  to  the  Organization  MOrt»r0p  crummy 
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Voice  of  the  Process 

Quantitative  Sub-Process  Management 


ASU  Log  Cost  Model 

Using  Lognormal  Probability  Density  Function 


5  - 


Review  Closed  Date 


Upper  Control  Limit 


Average  performance 


Lower  Control  Limit 


■  A  Stable  Process 

■  Operates  Within  the  Control  Limits  99.7%  of  the  Time 

■  Meets  Budget 

■  Offers  Opportunities  for  Systematic  Process  Improvement 
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Why  Peer  Reviews? 


■  Ubiquity 

■  Many  Work  Products  Reviewed  Throughout  Software 
Development  Life  Cycle 

■  Design  Artifacts 

■  Code 

■  Test  Plan,  Procedures  &  Reports 

■  Frequency 

■  High  Data  Rates 

■  Influence 

■  Approximately  1 0%  of  the  Software  Development  Effort 
Is  Spent  on  Peer  Reviews  and  Inspections 

■  Code  Walkthroughs  Represent  Biggest  Opportunity 
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Prior  State 

SW-CMM  Level  4 


■  Software  Development  Baseline  Characterized  by  Life 
Cycle  Phase 

■  SW  Requirements-Design-Code  &  Verification-SW 
Integration-System  Test 

■  10+  Year  Process  Improvement  Record  Resulted  in 
Costs  Reduced  by  Over  67% 

■  Lower  Level  Elements  Tracked  and  Managed  with 
Earned  Value  System 

■  No  “Above  the  Shop  Floor”  Experience  with  Statistical 
Sub-process  Control 

■  Issues  with  Peer  Review  Quality 

■  Inconsistent  Data 

■  Superficial  Results  crummy 
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Data  Characteristics 

Raw  Data 


Summary  for  Cost/LOC 


9  5%  Confidence  Intervals 


Median  - 


Probability  Plot  of  Cost/LOC 

Normal 


Andersen-Darling  Test  p  <  0.005 


Data  Non-normality  Violates 
Probability  Model 
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Can  Code  Walkthroughs  Be  Controlled? 


ASU  Cost  Model 

Raw  Data  for  Cost/LOC 
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Review  Closed  Date 


ASU  Eng  Chks  &  Elec  Mtngs 


■  Difficulties 

■  11%  False  Alarm  Rate  (Chebyshev’s  Inequality) 

■  Penalizes  Due  Diligence  in  Reviewing  Code 

■  No  Meaningful  Lower  Control  Limit 

■  Does  Not  Flag  Superficial  Reviews 

■  Arithmetic  Mean  Distorts  the  Central  Tendency 

■  Apparent  Cost  Will  Not  Meet  Budget 

NORTHROP  GRUMMAN 


AGS&BMS-PR-05-104 


Log -Return  Model 

Stock  Sales 


■  Consider  a  stock  sale  in  terms 
of  the  number  of  shares  sold 
for  a  certain  price 


■  The  natural  logarithm  of  the 
difference  between  the 
current  and  the  next  per  share 
sale  price  is  normally 
distributed  with  zero  mean 
and  a  constant  standard 
deviation 

■  Cost  basis 

■  $s  per  Share  Stock  Price 


Log -Cost  Model 

Peer  Reviews 


Consider  a  code  walkthrough 
in  terms  of  the  number  of 
lines  of  code  reviewed  in  a 
certain  number  of  hours 

By  analogy,  the  natural 
logarithm  of  the  difference  in 
cost  between  the  current  and 
the  next  peer  review  will  be 
normally  distributed  with  zero 
mean  and  a  constant  standard 
deviation 

Cost  Basis 

■  Hours  per  Line  of  Code 
Reviewed 
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Consequences _ 

Log-Return  Model 

Stock  Sales 

■  Stock  prices  themselves 
are  lognormally 
distributed 

■  The  natural  logarithms  of 
stock  prices  follow  a 
normal  distribution 

■  Thus,  the  log-return  data 
meet  the  assumptions 
needed  for  successful 
control  charting 


Log -Cost  Model 

Peer  Reviews 

■  Peer  review  costs  are 
lognormally  distributed 

■  The  natural  logarithms  of 
the  peer  review  costs 
follow  a  normal 
distribution 

■  Thus,  the  log-cost  data 
meet  the  assumptions 
needed  for  successful 
control  charting 
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Math  Details 


■  Consider  a  stochastic  process  . . .,  X2,  X. „  X0,  Xu  X# 
. . .  that  represents  an  asset  price  recorded  over 
time,  like  a  daily  sequence  of  prices  for  shares  of  a 
stock  or  other  commodity 

■  We  assume  at  time  t  that  the  realization  xt  of  Xt  is 
known,  but  the  realization  xt+1  of  Xt+1  is  unknown 

■  The  single-period  log-return,  /n(Xf+/xfj,  is  random 
and  assumed  to  be  normally  distributed,  at  the 
given  time  t 

■  Under  these  assumptions,  XM/xt  is  a  lognormally 
distributed  random  variable,  and  therefore,  so  is  XM 


Math  Details  extracted  from: 

http://www.riskglossarv.com/articles/lognormal  distribution.htm 
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Salient  Properties  of  the  Model 


■  When  log-returns  are  normally  distributed,  the 
corresponding  prices  are  lognormally  distributed 

■  This  model  “is  one  of  the  most  ubiquitous  models  in 
finance” 

■  The  distribution  of  log-returns  and  share  prices  have 
been  validated  empirically  by  many  market  studies 
accessible  on  the  web 

■  For  short  time  periods  in  a  stable  market,  the  mean 
return  is  0 


Quotation  from: 

http://www.riskglossary.com/articles/lognormal  distribution.htm 
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Lognormal  Density  Function 
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X~A[jU,cr2] 


Y  =  ln(X)  ~  7V[//,<x2] 


£(X)  =  exp(//  +  <x2/2) 


Var(X)  =  (exp(cr2)  - 1  )exp(2//  +  a2) 


Math  details  can  be  found  in  any  standard  mathematical  statistics  reference, 
see  for  example,  http://en.wikipedia.org/wiki/Lognormal  distribution. 
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Our  Data  on  Logs 


95%  Confidence  Intervals 


Mean- 


Median- 


ASU  Eng  Checks  &  Elec  Mtngs 


Probability  Plot  of  ln(COST/LOC) 

Normal 


In  (COST/LOC) 

ASU  Eng  Checks  &  Elec  Mtngs 


Andersen-Darling  Test  p  <  0.759 

A  Textbook  Demonstration 
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The  Transformed  Control  Chart 


ASU  Log  Cost  Model  -  Initial  Release 

Using  Lognormal  Probability  Density  Function 


Review  Closed  Date 


ASU  Eng  Checks  &  Elec  Mtngs 
Data  Through  Review  7351  on  8/25 

Impacts 

■  False  Alarms  Minimized 

■  Meaningful  Lower  Control  Limit 

■  Geometric  Mean  Preserves  the  Budget 

■  OK,  You  Still  Have  to  Find  the  Antilog 


An  In-control,  Stable 
Process 
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One  Year  Later  . . . 


■  Independent  Lead  Appraisers  Cited  Innovation  and 
Novelty  of  Log-cost  Model  in  Level  4  (10/2004)  and 
Level  5  (4/2005)  Appraisals 


ASU  Meeting  Peer  Review  Cost  Data  -  23SEP05 


30-Mar  12-May  10-J  un  28-J  un  29-J  ul  25-Aug  21- Sep  14-Oct  18-Oct  8-Nov 

Review  Closed  Date 
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Expanding  the  Capability 


■  Test,  SW  Design 


^  ASU  Physical  Model  Peer  Review  Cost  Data  -  23SEP05 

oT 

[g  2- Sep- 03 _  7-J  un-05 _ 


A 

m 


Date  Closed 


ASU  Test  Procedures  Peer  Review  Cost  Data  -  23SEP05 

12- Oct- 04  9- Dec- 04  10-Feb-05 


Review  Closed  Date 


ASU  Software  Threads  Peer  Review  Cost  Data  -  23SEP05 


11- Nov-03 


18-J  ul-05 
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Summary 


With  the  Log-cost  Model 

■  Peer  Review  Subprocesses  Are  In-control  and 
Capable  of  Meeting  Baseline  Budget  Allocations 

■  Due  Diligence  Is  Rewarded 

■  Superficial  Reviews  Are  Detected 

■  False  Alarm  Rate  Reduced 

■  Greater  Than  40x  Improvement 


Enhanced  Sub-Process  Control 
for  CMMI  Levels  4  and  5 
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QUESTIONS 


Richard  L.  W.  Welch,  PhD 

Northrop  Grumman  Corporation 
(321)  951-5072 
rick.welch@ngc.com 
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Agenda 

•  Continuous  Appraisal  Method  (CAM) 

•  SCAMPI  Preparation  Efficiencies 

•  Summary 


November  16,  2005 
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Continuous  Appraisal  Method  (CAM)  -  1 


Appraise  Organizational  Processes 


multiple  visits 


Incrementally  Appraise 
Organizational  Processes 


Appraise  Projects’  Processes 


Incrementally  Appraise 
3-4  Projects 


Conduct  Maintenance  Review 


Appraise  Additional  Projects 


v 


Incrementally  Appraise 
Additional  Projects  (Optional) 
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Continuous  Appraisal  Method  (CAM)  -  2 


Institutionalization  focus  with  minimal  project  disruption 
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Continuous  Appraisal  Method  (CAM)  -  3 


•  Integrate  process  improvement  with  appraisal 
preparation 

■  Weaknesses  are  fixed  by  the  organization  and  reappraised 
by  the  appraisal  team  during  the  course  of  the  appraisal 

■  Single  team  over  the  entire  process  improvement  and 
appraisal  preparation  effort 

•  Make  this  integrated  effort  less  expensive  and  less 
invasive  to  the  organization  and  projects 

•  Eliminate  rework  caused  by  rollout  of  organizational 
processes  with  weaknesses 

•  Focus  organizations  on  Continuous  Process 
Improvement  as  opposed  to  multiple  special  event 
“tests” 

•  Promote  institutionalization 
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SCAMPI  Preparation  Efficiencies 

•  Early  engagement  of  the  SCAMPI  Lead  Appraiser 

•  Synchronize  CAM  and  SCAMPI  schedules 

•  Preparation  of  Practice  Implementation  Indicator 
Descriptions  (PHD) 
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CAM  Planning  with  SCAMPI  Lead  Appraiser  -  1 


•  Engage  The  SCAMPI  Lead  Appraiser  at  the  start  of 
the  CAM 

•  The  SCAMPI  Lead  Appraiser  participates  in  CAM 
appraisal  planning  discussions  regarding: 

■  Organizational  scope 

■  Project  selection 

■  Model  scope 

■  Schedule 

■  Team  members 

■  Key  areas  of  method  and  model  interpretation 

■  Amount  of  direct  and  indirect  evidence  needed 
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CAM  Planning  with  SCAMPI  Lead  Appraiser  -  2 


•  Enables  agreement  on: 

■  Key  appraisal  parameters 

■  Interpretation  topics 

-  e.g. 

•  Validation 

•  GP  3.2  Collect  Improvement  Information 

■  Expectations  for  Practice  Implementation  Indicator 
Descriptions  (PHD)  content 


•  Document  agreements  between  Lead  Appraiser  and 
Sponsor  in  appraisal  plans 
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CAM  and  SCAMPI  Schedules  -  1 

•  Synchronize  the  CAM  and  SCAMPI  schedules 

•  Key  checkpoints  over  approximately  twelve  months 


•  Conduct  PHD  reviews 

■  Participants: 

-  Business  Unit  Site  Coordinator  (Lead) 

-  SCAMPI  Lead  Appraiser 

-  CAM  Lead  Appraiser 

-  One  internal  member  of  CAM  Appraisal  Team  (optional) 
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€M  CAM  and  SCAMPI  Schedules  -  2 


•  Notional  integrated  schedule 


Month 

Phase 

1  j 

2  I 

3 

4 

5 

6 

7 

8 

9 

10 

ii 

12 

13 

CAM  Plan  and  Prepare 

CAM  Appraise  OSP 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

■ 

i 

CAM  Appraise  PDP 

Maintenance  Review 

Engage  SCAMPI  Lead 

I 

Participate  in  CAM  Planning 

1 

Conduct  SCAMPI  PHD  Reviews^ 

'  V 

n 

OSP  &  Org  PAs  (1-2  Days) 

7 

Project  PAs  (3-5  Days) 

71 

1 

Maintenance  Review 

r 

J 

I/n 

l 

1 

Conduct  SCAMPI  Training  & 

Readiness  Review 

-A 

" 

J 

n 

Conduct  SCAMPI  Onsite  Event 

J 

L 

_ 

■ 

■ 

Synchronized  schedules  enable  agreement  between  CAM  and  SCAMPI  Lead 
Appraisers  and  the  organization  on  key  appraisal  parameters  and  outputs 


November  16,  2005 


©Copyright  2005  Lockheed  Martin  Corporation 


10 


Practice  Implementation  Indicator  Descriptions  -  1 


•  CAM  is  planned  and  conducted  to  support  creation  of 
a  complete  and  verified  PHD 


•  The  organization  prepares  a  PHD  which  demonstrates 
that  the  model  practices  are  both 

■  Required  by  the  organizational  processes 

■  Being  executed  by  the  programs 

•  More  cost  effective  for  the  organization  to  create  a 
draft  PHD  as  input  to  the  CAM 
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Practice  Implementation  Indicator  Descriptions  -  2 


•  CAM  team  verifies  PHD  and  documents  any  additional 
objective  evidence  discovered 


•  The  CAM  and  SCAMPI  Lead  Appraisers  agree  on  the 
adequacy  of  the  PHD  to  serve  as  an  input  to  the 
SCAMPI  via  multiple  reviews 


•  Reviews  conducted  after  CAM  activities: 

■  Organizational  Level  Appraisal  (phase  2) 

■  Project  Level  Appraisal  (phase  3) 

■  Maintenance  Review  (phase  4) 
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CAM  and  SCAMPI  Schedules  -  2 


•  Notional  integrated  schedule 


Month 

Phase 

1  j 

2  I 

3 

4 

5 

6 

7 

8 

9 

10 

ii 

12 

13 

CAM  Plan  and  Prepare 

CAM  Appraise  OSP 

1 

■ 

■ 

■ 

1 

■ 

■ 

1 

i 

CAM  Appraise  PDP 

Maintenance  Review 

Engage  SCAMPI  Lead 

■ 

Participate  in  CAM  Planning 

■ 

Conduct  SCAMPI  PHD  Reviews 

OSP  &  Org  PAs  (1-2  Days) 

Project  PAs  (3-5  Days) 

71 

1 

Maintenance  Review 

J 

I/n 

l 

1 

Conduct  SCAMPI  Training  & 

Readiness  Review 

-A 

" 

J 

n 

Conduct  SCAMPI  Onsite  Event 

J 

L 

_ 

■ 

■ 

PHD  reviews  with  CAM  and  SCAMPI  Lead  Appraisers  and  the  organization 


November  16,  2005 
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Summary 


•  Lockheed  Martin  CAM  has  been  used  successfully  to 
prepare  for  SCAMPI 

•  Reduces  risk  of  significant  weaknesses  being  found  in 
a  SCAMPI 

•  Supports  preparation/completion  and  validation  of 
SCAMPI  required  data  (PIID’s) 


November  16,  2005 
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Defining  the  future 
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Analyzing  Defects 


Can  Tell  a  Story 
About  a  Company 


1 
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CMMI  Technology  Conference  and  User  Group 

November  14-  17,  2005 


Diane  Mizukami  (Williams) 

Diane.Mizukami@nqc.com 

Northrop  Grumman  Corporation 


NORTHROP  GRUMMAN 

Agenda 
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4  Stages  of  Denial 


Arrogance 


We’re  perfect. 
We’re  a  fine 
tuned  machine. 
Analyzing  our 
defects  is  a 
waste  of  time. 


Resistance 


I  believe  you, 
but  we’ve 
survived  for 
years.  We 
don’t  need  to 
change. 
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Defensiveness 


We’re  not  perfect, 
but  I  don’t  believe 
your  analysis. 

Are  you  saying 
we’re 

incompetent? 


Skepticism 


We  really 
want  to 
improve,... 
except  for 
one  person. 


2 


11*.,. ng.-,  Hnnr*  i'r,| 
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Metrics  Takes  Patience,...  Don't  Give  Up 


71 
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You  might 


look  at  20  graphs  before  you  find  one  golden  nugget. 
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Try  Different  Graphs  to  Find  the  Story 
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Disaggregate  to  Find  the  Story 


NORTHROP  GRUMMAN 
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Introduction  to  Control  Charts 


What’s  the  average  minutes  from  home  to  the  LAX  gate? 
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Minutes 


NORTHROP  GRUMMAN 

Understand  Special  Causes 
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Use  Control  Charts  to  Make  Decisions 


NORTHROP  GRUMMAN 
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Minutes 
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Use  Control  Charts  to  Predict  the  Future 
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Real  Data  from  37  Trips 


20 
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Disaggregate  by  Components 
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Number  of  Defects  Found  Per  Week 


Disaggregate  by  Severity 


NORTHROP  GRUMMAN 


150 

100 

50 

0 

Severity 


Outlier 

<~-25% 

Median 

\ 

-*-25% 

Mean^"* 

-*-25% 

<-25% 

Box  Plots 

-X- 


* 


A  good  product 

The  developers’ 

delivered  to  test 

product  was  so 

typically  has  no 

* 

poor,  the  testers 

Severes,  a  few 

didn’t  have  time  to 

Highs,  and  some 

find  the  low  priority 

Mediums. 

nit-picky  problems. 

* 


t 

l==l 

- 1 - 

- 1 - 

0) 

_C 

<D 

O) 

> 

I 

TU 

0) 

CM 

0) 

(/) 

CO 

o 

_l 


Copyright  2005  Northrop  Grumman  Corporation 


11 


Total  Defects 


Disaggregate  by  Severity  and  Release 


NORTHROP  GRUMMAN 
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All  releases  had  a  similar  number  of 
Severe  problems,  i.e.,  no  shift  is 
observed;  therefore  Release  2.0’s 
quality  was  just  as  poor,  and  was 
probably  a  smaller  release. 
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Number  of  Defects  Found  Per  Week 


Dig  Deeper  for  Release  2.1 


NORTHROP  GRUMMAN 


200  - 


100 


0 


1 


UCL=1 12.2 


Mean=40.8 


LCL—30.54 


Copyright  2005  Northrop  Grumman  Corporation 


Week 


13 


Li  } 
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Disaggregate  by  Customer  vs  Test 
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Who  Finds  Defects?  Customer  or  Test? 


Test  for  Equal  Variance  to  check  if  variation  differs  between  groups 


0  50  100 

Number  of  Defects  Found  Per  Week 
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Number  of  Defects  Found 


Customer  vs  Test  Release  2.1 
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The  Story  (i  of  2) 


NORTHROP  GRUMMAN 


1 1k  The 
company  is 
aCMMI 
Level  -5 
company. 


I 


I 


CMIVLL  G*yel 

VJ 


21 


I  would 
never  buy 
their  poor 
quality 
product 


3  There  are  no 
processes, 
poor 

processes,  or 
engineers 
ignore 
processes 
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The  Story  (2  of  2) 


NORTHROP  GRUMMAN 


4  Engineers  are 
pressured  to 
deliver  before 
the  product  is 
ready 


5  Test  may  not 
be  at  fault; 
developers 
deliver  poor 
products  to 


test 


0  Customer 
complaints  will 
continue  until 
they  see 


CHANGE  and 
quality 
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Summary 


No  matter  what 
your  opinions 
are,  always 
analyze  defects. 
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You’ll  be  surprised  how 
much  you  can  find. 
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What  the  CM  MI 
Doesn't  Say  About 
Training 
(But  Should!) 


CMMI  Technology  Conference  &  User  Group 

14-17  November  2005 


Rick  Hefner,  Ph.D. 

Northrop  Grumman  Corporation 

Sree  Yellayi 

Siemens  Corporate  Research 


Agenda 


■  Key  questions  on  Organizational  Training  (OT)  capability 

■  Introduction  to  OT  in  CMMI 

■  CMMI  requirements  for  the  OT  process  area 

■  The  Kirkpatrick  model 

■  CMMI  requirements  for  the  Training  generic  practice 

■  Strategies  for  Organizational  Training 


CMMI  is  a  service  mark  of  Carnegie  Mellon  University 


NORTHROP  GRUMMAN 
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Key  Questions  in  Establishing 
an  Organizational  Training  Capability 


■  How  much  training  is  enough? 

■  Should  training  be  developed  in-house  or 
bought  from  a  vendor  or  university? 

■  How  should  training  be  paid  for? 

■  How  do  you  determine  whether  training  is  effective? 

■  How  does  the  training  needs  of  project  personnel,  staff  groups, 
and  management  differ? 

■  Are  alternatives  to  classroom  training  (informal  mentoring,  web- 
based  training,  guided  self-study,  on-the-job  training)  effective? 
Under  what  conditions? 

■  How  do  you  address  technical,  process,  organizational,  and 
contextual  knowledge? 


The  CMMI  provides  guidance  based  on  industry  best-practices 


SIEMENS 
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Organizational  Training  -  Level  3  Process  Area 


Level  Focus  Process  Areas 

5  Optimizing 

Continuous 

process 

improvement 

Causal  Analysis  and  Resolution 

Organizational  Innovation  and  Deployment 

4  Quantitatively 
Managed 

Quantitative 

management 

Quantitative  Project  Management 

Organizational  Process  Performance 

3  Defined 

Process 

standardization 

Organizational  Process  Focus 

Organizational  Process  Definition 

Organizational  Training 
integrated  Project  Management 

Risk  Management 

Decision  Analysis  and  Resolution 

Requirements  Development 

Technical  Solution 

Product  Integration 

Verification 

Validation 

Organizational  Environment  for  Integration 

Integrated  Teaming 

Integrated  Supplier  Management 

2  Managed 

Basic 

project 

management 

Requirements  Management 

Project  Planning 

Project  Monitoring  and  Control 

Supplier  Agreement  Management 

Measurement  and  Analysis 

Process  and  Product  Quality  Assurance 

Configuration  Management 

1  Performed 

SIEMENS 
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Organizational  Training  Process  Area 


■  Purpose 

■  Develop  the  skills  and  knowledge  of 
people  so  they  can  perform  their  roles 
effectively  and  efficiently 

■  Key  actions 

■  Identifying  the  training  needed  by  the  organization 

■  Obtaining  and  providing  training  to  address  those  needs 

■  Establishing  and  maintaining  training  materials 

■  Establishing  and  maintaining  training  records 

■  Assessing  training  effectiveness 


SIEMENS 
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Training  Scope 


1 


Skills  and  knowledge  may  be: 

■  Technical  -  ability  to  use  the  equipment,  tools, 
materials,  data,  and  processes 

■  Organizational  -  behavior  within  and  according  to  the 
employee's  organization  structure,  role  and 
responsibilities,  and  general  operating  principles  and 
methods 

■  Contextual  -  self  management,  communication,  and 
interpersonal  abilities  needed  to  successfully  perform  in 
the  organizational  and  social  context  of  the  project  and 
support  groups 


1 


■  Training  options 

■  Classroom  training 

■  Web-based  training 

■  Guided  self  study 

■  Formalized  on-the-job  mentoring 


SIEMENS 
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Is  the  Staff  Qualified  to  Do  Their  Work? 


What  are  the  minimum 
skills  and  knowledge 
needed  to  perform  their 
job  function? 

Does  each  individual 
possess  these  skills? 


An  organizational  responsibility! 


■  If  not,  training  is 
expected  to  address 
the  gaps 


How  does  the  organization  maintain 
a  skilled  and  knowledgeable  workforce? 


SIEMENS 
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Organizational  Training  Goals 


■  SG  1  Establish  an  Organizational  Training  Capability 
A  training  capability  that  supports  the  organization's 
management  and  technical  roles  is  established  and  maintained. 

■  SG  2  Provide  Necessary  Training 

Training  necessary  for  individuals  to  perform  their  roles 
effectively  is  provided. 

■  GG  3  Institutionalize  a  Defined  Process 

The  process  is  institutionalized  as  a  defined  process. 


NORTHROP  GRUMMAN 
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Expected  Practices  -  Goal  1  (Establish  an 
Organizational  Training  Capability) 


■  SP  1.1  Establish  the  Strategic  Training  Needs 
Establish  and  maintain  the  strategic  training 
needs  of  the  organization. 

■  SP  1 .2  Determine  Which  Training  Needs  Are  the 
Responsibility  of  the  Organization 
Determine  which  training  needs  are  the 
responsibility  of  the  organization  and  which  will 
be  left  to  the  individual  project 

or  support  group. 

■  SP  1.3  Establish  an  Organizational  Training 
Tactical  Plan 

Establish  and  maintain  an  organizational  training 
tactical  plan. 

■  SP  1.4  Establish  Training  Capability 
Establish  and  maintain  training  capability  to 
address  organizational  training  needs. 


■  Strategic  needs  address 
long-term  maintenance  of  a 
qualified  work  force 

■  Tactical  plans  address  this 
year’s  training 

■  The  organization  may 
choose  to  meet  some 
needs,  and  leave  other  to 
individual  projects 

■  Capability  includes 

■  Classrooms 

■  Training  materials 

■  Instructors 

■  Administrative  staff 
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Instructor  Qualification 


Qualify  Instructors 

■  Based  on  their  knowledge,  experience  and  skills 

■  Have  a  process  for  approving  instructors  before  they  start 
teaching 

■  Maintain  a  list  of  qualified  instructors 

■  Show  who  is  qualified  to  teach  which  course 

■  Update  the  list  to  include  new  instructors  from  time  to  time 

Re-train  the  instructors 
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Expected  Practices  -  Goal  2  (Provide  Necessary 
Training) 


■  SP  2.1  Deliver  Training 
Deliver  the  training  following  the 
organizational  training  tactical  plan. 

■  SP  2.2  Establish  Training  Records 
Establish  and  maintain  records  of  the 
organizational  training. 

■  SP  2.3  Assess  Training  Effectiveness 
Assess  the  effectiveness  of  the 
organization’s  training  program. 


Training  Feedback  form  taken  at  the  end 
of  training  course  is  not  measuring 
effectiveness 


■  The  purpose  of  training 
records  is  to: 

■  Determine  who  is 
qualified  for  each 
assignment 

■  Determine  how  many 
people  still  need  to 
take  each  required 
training  course  (drives 
budgets) 

■  Effectiveness  -  how  well 
did  the  training  impart  the 
desired  skills  and 
knowledge? 
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Effectiveness  —  The  Kirkpatrick  Model 


Level  1  - 
Collect 
student  and 
instructor 
reaction  to 
the  training 


Level  3  - 
Measure 

Level  2  - 

transference  of 

Measure 

learning  to  the 

student 

job 

learning 
through  testing 

Level  4  - 
Measure 
impact  on  job 
performance 
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Expected  Practices  —  Generic  Goal 
(Institutionalize  a  Defined  Process) 


■  GP  2.1  Establish  an  Organizational  Policy 

■  GP  2.2  Plan  the  Process 

■  GP  2.3  Provide  Resources 

■  GP  2.4  Assign  Responsibility 

■  GP  2.5  Train  People 

■  GP  2.6  Manage  Configurations 

■  GP  2.7  Identify  and  Involve  Relevant 
Stakeholders 

■  GP  2.8  Monitor  and  Control  the  Process 

■  GP  2.9  Objectively  Evaluate  Adherence 

■  GP  2.10  Review  Status  with  Higher  Level 
Management 

■  GP  3.1  Establish  a  Defined  Process 

■  GP  3.2  Collect  Improvement  Information 


"  Often  neglected  areas: 

■  Training  for  instructors 
and  administrative  staff 

■  Configuration  control  of 
course  materials, 
student  records 

■  Process  and  product 
audits 

■  Defined  processes  for 
needs  identification, 
student  selection, 
course  revisions,  etc. 
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Generic  Practice  for  Training 
(Applies  to  all  Process  Areas) 


■  GP  2.5  Train  People 

Train  the  people  performing  or  supporting  the  process  as 
needed. 


Requirements  Management 

Project  Planning 

Project  Monitoring  and  Control 

Supplier  Agreement  Management 

Measurement  and  Analysis 

Process  and  Product  Quality  Assurance 

Configuration  Management 

Requirements  Development 

Technical  Solution 

Product  Integration 

Verification 

Validation 

Organization  Process  Focus 

Organization  process  definition 

Organizational  Training 

Organizational  Environment  for  Integration 

Integrated  Teaming 

Integrated  Supplier  Management 

Integrated  Project  Management 

Risk  Management 

Decision  Analysis  and  Resolution 

Organizational  Process  Performance 

Quantitative  Project  Management 

Organizational  Innovation  and  Deployment 

Causal  Analysis  and  Resolution 

GP  2.1 

(COI) 

Establish  an  Organizational  Policy 

GP  2.2 

(AB1) 

Plan  the  Process 

GP  2.3 

(AB  2) 

Provide  Resources 

GP  2.4 

(AB  3) 

Assign  Responsibility 

► 

GP  2.5 

(AB  4) 

Train  People 

GP  3.1 

(AB  5) 

Establish  a  Defined  Process 

GP  2.6 

(DM) 

Manage  Configurations 

GP  2.7 

(DI2) 

Identify  and  Involve  Relevant  Stakeholders 

GP  2.8 

(DI3) 

Monitor  and  Control  the  Process 

GP  3.2 

(DI4) 

Collect  Improvement  Information 

GP  2.9 

(VE1) 

Objectively  Evaluate  Adherence 

GP  2.1 0 

(VE2) 

Review  Status  with  Higher-Level  Management 

X. 
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Strategies  for  Organizational  Training  - 1 


■  Start  by  defining  the  key  job  functions  in 
the  organization 

■  E.g.,  project  manager,  software  engineer, 
quality  assurance  specialist 

-  Identify  the  requisite  knowledge  associated 
with  each  function 

-  Define  a  set  of  course  modules  that  impart 
this  knowledge 

■  Map  modules  to  job  functions 


■  Some  modules  will  be  common  to  multiple  job  functions 


■  Acquire  training  materials  and  trainers 

■  Should  reflect  the  organization’s  policies  and  processes 

■  Unlikely  that  standard  vendor/university  courses  will  fit 


Ensure  all  the  CMMI  process  areas  are  addressed 

■  Knowledge  needed  to  perform  the  process,  NOT  a  course  about  the  CMMI 
requirements  for  that  process  area 

■  Include  performers  of  the  process,  and  those  that  support  the  process 

NORTHROP  GRUMMAN 
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Strategies  for  Organizational  Training  -  2 


■  Identify  each  employee  by  their  job  function(s),  map  to  required 
courses 

■  If  the  employee  already  has  the  identified  minimum  knowledge, 
they  do  not  need  to  take  the  course 

■  Establish  student  records 

■  Who  has  completed  what  course,  waivers 


Review  required  training  with  employees 

■  Career-planning,  promotions,  new  hires 

Where  additional  project-specific  training 
is  required  (e.g.,  tools,  methods),  adopt  a 
similar  approach  at  the  project  level 

■  Project  Planning  SP  2.5 
addresses  project  specific  training 
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Objectives 


■  Describe  BAE  Systems  Information  Technology’s 
(BAE-IT’s)  journey  implementing  CMMI  Level  3  in  a 
services  environment  for  Defense  and  National 
customers 

■  Offer  insight  into  the  management  challenges  BAE-IT 
encountered  during  the  implementation 

■  Share  BAE-IT’s  Lessons  Learned 
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Introduction  to  BAE-IT 

-  Who  is  BAE-IT? 

-  Mission 

-  Breadth  of  services 

»  IT  services 

»  Mission  Support  services 

-  Organizational  structure 

-  BAE-IT’s  CMMI  Journey 

-  Experience  implementing  CMMI  Level  3  in  a  services 
environment 

-  Nature  of  challenges 

-  Resolution  processes 
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Challenges  Aggregated  into  8  Primary  Foci 

■  Senior  Management 

■  Communications 

■  Budget 

■  Resources 

■  Schedule 

■  Security 

■  Customer 

■  Project  Management 
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Senior  Management 

■  Challenge 

-  To  involve  Senior  Management,  gain  stakeholder  buy-in, 
and  obtain  commitment  to  the  goal  -  using  a  process 
improvement  methodology  typically  associated  with  SE/SW 

■  Mitigation  Strategies 

-  Management  structure 

»  Office  of  Performance  Excellence 

-  Authority 

-  Management  buy-in 

-  Emphasize  value-add 
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Communications 

■  Challenge 

-  To  ensure  effective  and  timely  stakeholder  communication 
across  geographically  diverse  locations  with  limited 
electronic  access 

■  Mitigation  Strategies 

-  Plan 

-  Schedule 

-  Types/Media/Venues 

-  Across  the  enterprise 

-  Senior  management  updates 

-  Project  management  updates 

-  ATL  involvement 

-  Issues 
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Budget 

■  Challenge 

-  To  gain  corporate  commitment  to  a  budget,  with  the  flexibility 
to  adjust  priorities,  in  a  services  environment  that  is  primarily 
LOE-based 

■  Mitigation  Strategies 

-  Corporate  resources 

-  Project  full-time  equivalent  staff  (FTEs) 

-  Training 

-  Consultant  support 

-  Cost  control 

-  Tools 
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Resources 

■  Challenge 

-  To  hire  and  retain  CMMI  model-knowledgeable, 
implementation-experienced,  resources  as  a  support 
function  in  a  services-driven  business 

■  Mitigation  Strategies 

-  Model-knowledgeable 

-  Implementation  experienced 

-  Understand  business  model 

-  Consultants  in  a  non-traditional  role 

-  Utilization/stress 

-  Adjusting/correcting  resources 

-  Adjusting/correcting  tasks/task  emphasis 
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Schedule 

■  Challenge 

-  To  work  to  an  externally  imposed  schedule  deadline  and 
implement  to  business  priorities 

■  Mitigation  Strategies 

-  Detailed  project  schedule  with  WBS  Dictionary 

-  Model  knowledge 

-  Documentation 

-  Stakeholder  buy-in 

-  Customer  constraints  addressed 

-  Institutionalization 
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Security 

■  Challenge 

-  To  provide  appropriate  access  to  secure  data  of  projects 
performing  services  across  multiple  secure  sites,  often  co¬ 
located  with  the  customer 

■  Mitigation  Strategies 

-  Project/site  information  and  access 

-  PAL  access 

-  PPQA  audit  activity 

-  Artifacts 

-  PHD  access 

-  Appraisal  participation 
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Customer 

■  Challenge 

-  To  gain  customer  acceptance  for  implementing  CMMI  in  a 
services  environment 

■  Mitigation  Strategies 

-  Emphasize  value-add  of  CMMI  for  services 

-  Level  of  involvement 

-  Communications 

-  Training 
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Project  Management 


■  Challenge 

-  To  structure  the  organizational  resources  best  to  provide 
direction,  validate  model  interpretation,  and  ensure  project 
success  on  services  contracts 

■  Mitigation  Strategies 

-  Utilizing  Project  Management  structure/format  for  CMMI 
initiative 

-  Treating  projects  as  customers 

-  Model  interpretation 

-  Hard  schedule  deadline 
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Lessons  Learned  -- 


1.  Senior  management  buy-in  and  ongoing  support 
were  highly  visible  and  crucial  to  ensuring 
corporate  commitment 

2.  Appropriate  Responsibilities,  Accountabilities,  and 
Authorities  (RAAs)  must  be  assigned  to  lead  the 
CMMI  initiative 

3.  CMMI  model-knowledgeable  and  implementation- 
experienced  staff  were  key 

4.  Adequate  funding/FTEs  at  HQ  and  project  levels 
were  required  for  success 

5.  A  detailed  Project  Plan  lent  credibility  to  CMMI 
initiative  and  helped  earn  stakeholder  buy-in 
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Lessons  Learned  -- 


6.  Stakeholder  communication  gained  necessary 
support 

7.  ATM  access  to  sensitive  project  sites/data  was 
required  for  complete  and  full  appraisal  of  process 
maturity 

8.  Non-traditional  use  of  consultants  as  full-time,  on¬ 
site  project  team  members  provided  key  focus  & 
direction 

9.  Early  focus  on  alignment  of  PAs  against  business 
priorities  &  resources  ensured  a  CMMI 
implementation  consistent  with  business  focus 

10.  Adopting  a  project  management  structure  for  the 
CMMI  initiative  elevated  the  probability  of  success 
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Lessons  Learned  -- 


11.  Tool  selection/investment  must  be  commensurate 
with  business  needs 

12.  An  early  working  relationship  with  the  ATL  afforded 
insight  into,  and  feedback  on,  our  implementation 
approach  and  Model  interpretation 

13.  Incorporating  a  QPM  focus  in  ongoing  efforts, 
keyed  to  services  key  measurement  indicators, 
provided  an  important  baseline  for  maturing  the 
organization 

14.  The  mapping  of  the  CMMI  Model  to  a  services 
environment  illustrated  the  flexibility  and 
tailorability  of  the  model  and  its  clear  application  to 
our  business 
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Conclusions  &  Going  Forward 


■  The  CMMI  Model  is  flexible  and  provides  documented 
benefits  in  a  services  environment 

■  BAE-IT’s  business  is  growing  and  continued  focus  on 
interpreting  the  CMMI  Model  for  our  services  business 
will  be  necessary 

■  We  must  structure  our  future  approach  to  address  our 
key  business  issues  and  strategic  goals  as  a  services 
provider 

■  BAE-IT  will  implement  ITIL  (Information  Technology 
Infrastructure  Library)  consistent  with  the  CMMI  Model 

■  SCAMPI  certification  of  maturity  against  the  CMMI 
Model  will  continue  to  be  a  discriminator  in  our 
services  industry 
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Contact  Information 


-  Thomas  E.  Zience 

Managing  Director,  Office  of  Performance  Excellence 

BAE  Systems  Information  Technology 

8201  Greensboro  Drive,  12th  Floor 

McLean,  VA  22102 

Office:  703.847.5820 

Fax:  703.847.5880 


■  Roger  W.  Lee 

Director,  Business  Process  Improvement 
BAE  Systems  Information  Technology 
8201  Greensboro  Drive,  12th  Floor 
McLean,  VA  22102 
Office:  703.873.3762 
Fax:  703.847.5880 
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